Communication management apparatus, image communication system, communication management method, and recording medium

ABSTRACT

A communication management apparatus for communicating with a communication terminal, including circuitry that receives from the communication terminal pieces of designated-position information transmitted, each indicating a corresponding one of a plurality of designated positions in an area where the plurality of image capturing devices are provided, and transmits to the communication terminal identification information of a shareable image capturing device and pieces of predetermined-area information of the shareable image capturing device, the shareable image capturing device being an image capturing device for which distances from any two designated positions among the plurality of designated positions indicated by the pieces of designated-position information are shortest and which is capable of sharing images of the two designated positions, each of the pieces of predetermined-area information of the shareable image capturing device indicating a predetermined area in which a corresponding one of the two designated positions is viewed at the front.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application No. 2020-166241, filed onSep. 30, 2020, in the Japan Patent Office, the entire disclosure ofwhich is hereby incorporated by reference herein.

BACKGROUND Technical Field

The present disclosure relates to a communication management apparatus,an image communication system, a communication management method, and anon-transitory recording medium.

Description of the Related Art

An image capturing device capable of capturing an omnidirectional sceneusing a plurality of wide-angle lenses or fish-eye lenses is known. Alsoknown is a system capable of distributing image data of an imagecaptured using such an image capturing device in real time such that adistribution site where the image capturing device is disposed isviewable at a different site in real time.

For example, there is a system in which images captured by a pluralityof cameras at a remote location are displayed on a terminal such that auser can grasp the situation at the remote location.

In the existing method, however, it is difficult to simultaneouslyprovide captured images of a plurality of points of interest using asingle image capturing device.

SUMMARY

Example embodiments include a communication management apparatus forcommunicating with a communication terminal that displays imagescaptured by a plurality of image capturing devices. The communicationmanagement apparatus includes circuitry that receives pieces ofdesignated-position information transmitted from the communicationterminal, each of the pieces of designated-position informationindicating a corresponding one of a plurality of designated positions inan area where the plurality of image capturing devices are provided. Thecircuitry further transmits, to the communication terminal,identification information of a shareable image capturing deviceselected from among the plurality of image capturing devices and piecesof predetermined-area information of the shareable image capturingdevice, the shareable image capturing device being an image capturingdevice for which distances from any two designated positions among theplurality of designated positions indicated by the pieces ofdesignated-position information that are received are shortest and whichis capable of sharing images of the two designated positions, each ofthe pieces of predetermined-area information of the shareable imagecapturing device indicating a predetermined area in which acorresponding one of the two designated positions is viewed at thefront.

Example embodiments include an image communication system including thecommunication management apparatus, and the communication terminal.

Example embodiments include a communication management method, performedby a communication management apparatus, the communication managementapparatus being configured to communicate with a communication terminalthat displays images captured by a plurality of image capturing devices.The communication management method includes: receiving pieces ofdesignated-position information transmitted from the communicationterminal, each of the pieces of designated-position informationindicating a corresponding one of a plurality of designated positions inan area where the plurality of image capturing devices are provided; andtransmitting to the communication terminal identification information ofa shareable image capturing device selected from among the plurality ofimage capturing devices and pieces of predetermined-area information ofthe shareable image capturing device, the shareable image capturingdevice being an image capturing device for which distances from any twodesignated positions among the plurality of designated positionsindicated by the received pieces of designated-position information areshortest and which is capable of sharing images of the two designatedpositions, each of the pieces of predetermined-area information of theshareable image capturing device indicating a predetermined area inwhich a corresponding one of the two designated positions is viewed atthe front.

Example embodiments include a non-transitory recording medium which,when executed by one or more processors, causes the processors toperform the communication management method.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIGS. 1A, 1B, and 1C are a side view, a front view, and a plan view ofan image capturing device according to an embodiment of the presentdisclosure, respectively;

FIG. 2 is an illustration for explaining how a user uses the imagecapturing device, according to an embodiment of the present disclosure;

FIGS. 3A, 3B, and 3C are views illustrating a hemispherical image (frontside) captured by the image capturing device, a hemispherical image(back side) captured by the image capturing device, and an image inequirectangular projection, respectively, according to an embodiment ofthe present disclosure;

FIG. 4A is a conceptual diagram illustrating how a surface of a sphereis covered with the equirectangular projection image according to anembodiment of the present disclosure;

FIG. 4B is a view illustrating a spherical image according to anembodiment of the present disclosure;

FIG. 5 is a view illustrating positions of a virtual camera and apredetermined area in a case in which the spherical image is of athree-dimensional solid sphere according to an embodiment of the presentdisclosure;

FIG. 6A is a perspective view of the virtual camera and thepredetermined area illustrated in FIG. 5 according to an embodiment ofthe present disclosure;

FIG. 6B is a view illustrating an image of the predetermined areadisplayed on a display according to an embodiment of the presentdisclosure;

FIG. 7 is a view illustrating a relationship between predetermined-areainformation and the image of the predetermined area according to anembodiment of the present disclosure;

FIG. 8 is a view illustrating a point in a three-dimensional Euclideanspace according to spherical coordinates, according to an embodiment ofthe present disclosure;

FIG. 9 is a diagram illustrating the general arrangement of an imagecommunication system according to an embodiment of the presentdisclosure;

FIG. 10 is a view illustrating an overview of a distribution site in theimage communication system according to an embodiment of the presentdisclosure;

FIG. 11 is a diagram illustrating an example hardware configuration ofthe image capturing device according to an embodiment of the presentdisclosure;

FIG. 12 is a diagram illustrating an example hardware configuration of adistribution terminal, a communication management apparatus, and acommunication terminal according to an embodiment of the presentdisclosure;

FIG. 13 is a diagram illustrating an example functional configuration ofthe image communication system according to an embodiment of the presentdisclosure;

FIG. 14 is a diagram illustrating an example functional configuration ofthe image communication system according to an embodiment of the presentdisclosure;

FIG. 15A is a conceptual diagram illustrating an example image capturingdevice management table according to an embodiment of the presentdisclosure;

FIG. 15B is a conceptual diagram illustrating an example image typemanagement table according to an embodiment of the present disclosure;

FIG. 16A is a conceptual diagram illustrating an example sessionmanagement table according to an embodiment of the present disclosure;

FIG. 16B is a conceptual diagram illustrating an example image typemanagement table according to an embodiment of the present disclosure;

FIG. 17 is a conceptual diagram illustrating an examplepredetermined-area management table according to an embodiment of thepresent disclosure;

FIG. 18 is a conceptual diagram illustrating an example arrangementinformation management table according to an embodiment of the presentdisclosure;

FIG. 19 is a conceptual diagram illustrating an exampledistribution-site management table according to an embodiment of thepresent disclosure;

FIG. 20A is a conceptual diagram illustrating an example image typemanagement table according to an embodiment of the present disclosure;

FIG. 20B is a conceptual diagram illustrating an examplepredetermined-area management table according to an embodiment of thepresent disclosure;

FIG. 21 is a view schematically describing an example relationshipbetween two points of interest and image capturing devices, according toan embodiment of the present disclosure;

FIG. 22 is a view schematically describing an example of installationdirections and azimuth angles of image capturing devices near a firstpoint of interest, according to an embodiment of the present disclosure;

FIG. 23 is a view schematically describing an example of installationdirections and azimuth angles of image capturing devices near a secondpoint of interest, according to an embodiment of the present disclosure;

FIG. 24 is a view schematically describing an example relationshipbetween three points of interest and image capturing devices, accordingto an embodiment of the present disclosure;

FIG. 25 is a view schematically describing an example relationshipbetween points of interest and image capturing devices arranged inirregular positions, according to an embodiment of the presentdisclosure;

FIG. 26 is a sequence diagram illustrating an example process forparticipating in a specific communication session in the imagecommunication system according to an embodiment of the presentdisclosure;

FIG. 27 is a view illustrating an example screen for selecting acommunication session according to an embodiment of the presentdisclosure;

FIG. 28 is a sequence diagram illustrating an example process formanaging image type information in the image communication systemaccording to an embodiment of the present disclosure;

FIG. 29 is a sequence diagram illustrating an example process forcommunicating captured image data and audio data in the imagecommunication system according to an embodiment of the presentdisclosure;

FIG. 30 is a sequence diagram illustrating an example process fordisplaying points of interest in the image communication systemaccording to an embodiment of the present disclosure;

FIG. 31 is a flowchart illustrating an example process for selecting animage capturing device according to an embodiment of the presentdisclosure;

FIG. 32 is a flowchart illustrating an example process for selecting ashareable image capturing device according to an embodiment of thepresent disclosure;

FIG. 33 is a view illustrating an example point-of-interest designationscreen displayed on the communication terminal according to anembodiment of the present disclosure;

FIG. 34 is a view illustrating an example display screen on which animage captured by an image capturing device near the first point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 35 is a view illustrating an example display screen on which animage captured by an image capturing device near the first point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 36 is a view illustrating an example display screen on which animage captured by an image capturing device near the first point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 37 is a view illustrating an example display screen on which animage captured by an image capturing device near the first point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 38 is a view illustrating an example display screen on which animage captured by an image capturing device near the second point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 39 is a view illustrating an example display screen on which animage captured by an image capturing device near the second point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 40 is a view illustrating an example display screen on which animage captured by an image capturing device near the second point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 41 is a view illustrating an example display screen on which animage captured by an image capturing device near the second point ofinterest is displayed, according to an embodiment of the presentdisclosure; and

FIG. 42 is a view illustrating an example display screen on thecommunication terminal in response to selection of a plurality of pointsof interest, according to an embodiment of the present disclosure; and

FIG. 43 is a sequence diagram illustrating a modification of the processfor displaying points of interest in the image communication system.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted. Also, identical or similar referencenumerals designate identical or similar components throughout theseveral views.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Referring now to the drawings, embodiments of the present disclosure aredescribed below. As used herein, the singular forms “a,” “an,” and “the”are intended to include the plural forms as well, unless the contextclearly indicates otherwise.

Hereinafter, one or more embodiments of the present disclosure will bedescribed with reference to the drawings. In the drawings, any redundantdescriptions thereof will be omitted.

Image Generation Method

An image generation method according to one or more embodiments will bedescribed with reference to FIGS. 1A to 8.

First, the external appearance of an image capturing device 10 will bedescribed with reference to FIGS. 1A to 1C. The image capturing device10 is a digital camera for capturing images from which a 360-degreespherical image is generated. FIGS. 1A, 1B, and 1C are a side view, afront view, and a plan view of the image capturing device 10,respectively.

As illustrated in FIG. 1A, the image capturing device 10 has a size suchthat a person can hold the image capturing device 10 with one hand. Asillustrated in FIGS. 1A, 1B, and 1C, the image capturing device 10includes an imaging element 103 a and an imaging element 103 b in anupper portion thereof such that the imaging element 103 a is disposed onthe front side and the imaging element 103 b is disposed on the backside. The imaging elements (image sensors) 103 a and 103 b are usedtogether with an optical member (e.g., lenses 102 a and 102 b describedbelow) capable of capturing a hemispherical image (with an angle of viewof 180 degrees or more). As illustrated in FIG. 1B, the image capturingdevice 10 further includes an operation device 115 such as a shutterbutton on the back surface of the image capturing device 10.

Next, a situation in which the image capturing device 10 is used will bedescribed with reference to FIG. 2. FIG. 2 is an illustration forexplaining how a user uses the image capturing device 10. As illustratedin FIG. 2, the image capturing device 10 is used to, for example,capture an image of objects around the user when the user holds theimage capturing device 10 with one hand. The imaging elements 103 a and103 b illustrated in FIGS. 1A to 1C capture the objects surrounding theuser to obtain two hemispherical images.

Next, an overview of processing for images captured by the imagecapturing device will be described with reference to FIGS. 3A to 4B.FIG. 3A is a view illustrating a hemispherical image (front side)captured by the image capturing device 10, FIG. 3B is a viewillustrating a hemispherical image (back side) captured by the imagecapturing device 10, and FIG. 3C is a view illustrating an image inequirectangular projection (hereinafter referred to as “equirectangularprojection image”). FIG. 4A is a conceptual diagram illustrating how asurface of a sphere is covered with the equirectangular projectionimage, and FIG. 4B is a view illustrating a spherical image.

As illustrated in FIG. 3A, an image obtained by the imaging element 103a is a curved hemispherical image (front side) captured through the lens102 a described below. Also, as illustrated in FIG. 3B, an imagecaptured by the imaging element 103 b is a curved hemispherical image(back side) captured through the lens 102 b described below. The imagecapturing device 10 combines the hemispherical image (front side) andthe hemispherical image (back side), which is flipped by 180 degrees, tocreate an equirectangular projection image EC illustrated in FIG. 3C.

Then, as illustrated in FIG. 4A, the image capturing device 10 uses OpenGraphics Library for Embedded Systems (OpenGL ES) to map theequirectangular projection image EC onto the surface of the sphere so asto cover the surface of the sphere to create a spherical image(spherical panoramic image) CE illustrated in FIG. 4B. In other words,the spherical image CE is represented such that the equirectangularprojection image EC corresponds to a surface facing the center of thesphere. OpenGL ES is a graphics library used for visualizingtwo-dimensional (2D) and three-dimensional (3D) data. The sphericalimage CE may be either a still image or a moving image.

As described above, the spherical image CE is an image mapped onto asphere surface so as to cover the sphere surface and may look strange tothe human eye. The image capturing device 10 displays the sphericalimage CE such that a predetermined area T included in the sphericalimage CE is represented as a flat image having fewer curves. Thus, it isless likely that a person viewing the spherical image CE feels strange.An image of the predetermined area T is hereinafter referred to as“predetermined-area image”. The display of the predetermined-area imagewill be described with reference to FIGS. 5 to 8.

FIG. 5 is a view illustrating positions of a virtual camera IC and thepredetermined area T in a case in which the spherical image CE is of athree-dimensional solid sphere. The virtual camera IC is at a positionof a point of view of a user who is viewing the spherical image CEdisplayed on the surface of the three-dimensional solid sphere. FIG. 6Ais a perspective view of the virtual camera IC and the predeterminedarea T illustrated in FIG. 5, and FIG. 6B is a view illustrating animage of the predetermined area T displayed on a display. In FIG. 6A,the spherical image CE illustrated in FIG. 5 is represented by athree-dimensional solid sphere CS. Assuming that the spherical image CEgenerated in the way described above is represented by the solid sphereCS, the virtual camera IC is located inside the spherical image CE, asillustrated in FIG. 5. The predetermined area T in the spherical imageCE is an imaging area of the virtual camera IC and is specified bypredetermined-area information indicating an imaging direction and anangle of view of the virtual camera IC in a three-dimensional virtualspace including the spherical image CE. Zooming of the predeterminedarea T may be represented by moving the virtual camera IC toward or awayfrom the spherical image CE. A predetermined-area image Q is an image ofthe predetermined area T in the spherical image CE. Accordingly, thepredetermined area T can be specified by an angle of view α and adistance f from the virtual camera IC to the spherical image CE (seeFIG. 7).

The predetermined-area image Q illustrated in FIG. 6A is displayed on apredetermined display as an image of the imaging area of the virtualcamera IC, as illustrated in FIG. 6B. The image illustrated in FIG. 6Bis a predetermined-area image represented by predetermined-areainformation that is set by default. In the following, a description willbe given using the imaging direction (ea, aa) and the angle of view (a)of the virtual camera IC. The predetermined area T may be indicated by,instead of the angle of view a and the distance f, the imaging area (X,Y, Z) of the virtual camera IC, which is the predetermined area T.

Next, a relationship between the predetermined-area information and theimage of the predetermined area T will be described with reference toFIG. 7. FIG. 7 is a view illustrating a relationship between thepredetermined-area information and the image of the predetermined areaT. As illustrated in FIG. 7, “ea” denotes an elevation angle, “aa”denotes an azimuth angle, and “α” denotes an angle of view of thevirtual camera IC. The position of the virtual camera IC is changed suchthat the point of gaze of the virtual camera IC, which is indicated bythe imaging direction (ea, aa), matches a central point CP(x, y) of thepredetermined area T serving as the imaging area of the virtual cameraIC. As illustrated in FIG. 7, the central point CP(x, y) of thepredetermined area T, whose diagonal angle of view is represented by theangle of view α of the virtual camera IC and is denoted by α, is used asa parameter (x, y) of the predetermined-area information. Thepredetermined-area image Q is an image of the predetermined area T inthe spherical image CE. The distance from the virtual camera IC to thecentral point CP(x, y) of the predetermined area T is denoted by “f”.The distance between the center point CP and a given vertex of thepredetermined area T is denoted by “L” (2L is a diagonal line). In FIG.7, a trigonometric function equation typically expressed by Equation (1)below is satisfied.

L/f=tan(α/2)  (1)

The image capturing device 10 described above is an example of an imagecapturing device capable of acquiring a wide-angle view image, and thespherical image CE is an example of the wide-angle view image. Thewide-angle view image is typically an image captured using a wide-anglelens, such as a lens capable of capturing an image of a wider range thanthat the human eye can perceive. Further, the wide-angle view image istypically an image taken with a lens having a focal length of 35 mm orless in terms of 35 mm film.

FIG. 8 is a view illustrating a point in a three-dimensional Euclideanspace according to spherical coordinates. The position coordinates ofthe central point CP when expressed by the spherical polar coordinatesystem is defined as (r, θ, ϕ). The coordinates (r, θ, ϕ) represent theradial distance, the polar angle, and the azimuth angle, respectively.The radial distance r is a distance from the origin of athree-dimensional virtual space including the spherical image CE to thecentral point CP and is thus equal to the distance f. FIG. 8 illustratesa relationship among the radial distance r, the polar angle θ, and theazimuth angle ϕ. In the following, a description will be given using theposition coordinates (r, θ, ϕ) of the virtual camera IC.

Overview of Image Communication System

Next, an overview of an image communication system according to thisembodiment will be described with reference to FIG. 9. FIG. 9 is adiagram illustrating an example of the general arrangement of an imagecommunication system 1. The image communication system 1 illustrated inFIG. 9 is a system in which captured images such as video imagesdistributed from a plurality of distribution sites are displayed at aplurality of viewing sites to provide real-time viewing of wide-rangeimages (e.g., spherical images) obtained by capturing scenes at thedistribution sites. In this embodiment, the distribution site is alsoreferred to as “area”.

As illustrated in FIG. 9, the image communication system 1 includesimage capturing devices 10 (image capturing devices 10A and 10B) anddistribution terminals 30 (distribution terminals 30A and 30B) locatedat a plurality of distribution sites (distribution site A (area A) anddistribution site B (area B)), a communication management apparatus 50,and communication terminals 70 (communication terminals 70C and 70D)located at a plurality of viewing sites (viewing sites C and D). Theimage capturing devices 10A and 10B are hereinafter referred tocollectively as “image capturing devices 10” or individually as “imagecapturing device 10” unless distinguished. The distribution terminals30A and 30B are hereinafter referred to collectively as “distributionterminals 30” or individually as “distribution terminal 30” unlessdistinguished. The communication terminals 70C and 70D are hereinafterreferred to collectively as “communication terminals 70” or individuallyas “communication terminal 70” unless distinguished.

The distribution terminals 30, the communication management apparatus50, and the communication terminals 70 of the image communication system1 can communicate with each other via a communication network 100. Thecommunication network 100 is constructed by the Internet, a mobilecommunication network, a local area network (LAN), or the like. Thecommunication network 100 may include, in addition to a wiredcommunication network, a network based on a wireless communicationstandard such as third generation (3G), fourth generation (4G), fifthgeneration (5G), Wireless Fidelity (Wi-Fi) (Registered Trademark),Worldwide Interoperability for Microwave Access (WiMAX), or Long TermEvolution (LTE).

As described above, the image capturing device 10 is a special digitalcamera configured to capture an image of an object or surroundings suchas scenery to obtain two hemispherical images, from which a sphericalimage is generated. The captured image obtained by the image capturingdevice 10 may be a moving image or a still image or may include both amoving image and a still image. Further, the captured image may includean image and audio. The distribution terminal 30 is configured toacquire an image from the image capturing device 10 via a wired cablesuch as a Universal Serial Bus (USB) cable and distribute the acquiredimage to the communication terminal 70 via the communication managementapparatus 50. In one example, the image capturing device 10A and thedistribution terminal 30A are located at the same site, namely, thedistribution site A. The image capturing device 10B and the distributionterminal 30B are located at the same site, namely, the distribution siteB. The number of distribution sites used is not limited to two, and onedistribution site or three or more distribution sites may be used. Inaddition, the image capturing device 10 and the distribution terminal 30may be connected wirelessly using short-range wireless communication orthe like, instead of using a wired cable.

The communication management apparatus 50 controls communication betweenthe distribution terminals 30 and the communication terminals 70 andmanages types of image data (e.g., general image and special image) tobe transmitted and received. In one example, the special image is aspherical image. The communication management apparatus 50 is arrangedin a service company or the like that provides image communicationservices. The communication management apparatus 50 has a serverfunction and provides identification information of image data (imagedata ID), the IP address of an image capturing device, image typeinformation representing the type of the image data, and the like inresponse to a request from the communication terminal 70.

The communication management apparatus 50 may be constructed by a singlecomputer or a plurality of computers that are assigned to dividedcomponents (functions) as appropriate. All or some of the functions ofthe communication management apparatus 50 may be implemented by a servercomputer existing in a cloud environment or a server computer existingin an on-premise environment.

The communication terminal 70 is a computer such as a personal computer(PC), which is used by a user at each viewing site. The communicationterminal 70 displays an image (a still image and/or a moving image)distributed from the distribution terminal 30. The communicationterminal 70 acquires a spherical image, which is an image captured bythe image capturing device 10, via the communication network 100. Thecommunication terminal 70 has installed therein OpenGL ES and is capableof creating predetermined-area information indicating a partial area ofa spherical image sent from the distribution terminal or creating apredetermined-area image from the spherical image. In one example, thecommunication terminal 70C is placed at the viewing site C where a userC1 is located, and the communication terminal 70D is placed at theviewing site D where a user D1 is located.

The arrangement of the terminals and devices (i.e., the communicationterminals 70, the image capturing devices 10, and the distributionterminals 30) and the users C1 and D1 illustrated in FIG. 9 is anexample, and another example may be used. Each of the communicationterminals 70 is not limited to a PC and may be, for example, a tabletterminal, a smartphone, a wearable terminal, a projector (PJ), anInteractive White Board (IWB), which is an electronic whiteboard withmutual communication capability, a telepresence robot, or the like.

A distribution site in the image communication system 1 will now beschematically described with reference to FIG. 10. FIG. 10 is a viewillustrating an overview of a distribution site in the imagecommunication system 1. FIG. 10 illustrates the image capturing devices10A and the distribution terminal 30A, which are arranged at thedistribution site A. In FIG. 10, a plurality of vehicles are arrangedbetween the plurality of image capturing devices 10A, when viewed fromabove. While FIG. 10 illustrates an example of the distribution site A,the same applies to other distribution sites such as the distributionsite B.

The distribution site A illustrated in FIG. 10 is, for example, a spacesuch as an automobile showroom, in which a plurality of image capturingdevices 10A and a distribution terminal 30A capable of communicatingwith the image capturing devices 10A are arranged. The distribution siteis not limited to the automobile showroom and may be any space for whicha user (or viewer) at a viewing site desires to remotely grasp thesituation, and examples of the distribution site include an officefloor, a school, a factory, a warehouse, a construction site, a serverroom, and a store. For example, the image capturing devices 10A arearranged at predetermined intervals on pillars arranged in the showroomor on the top of poles or other objects having a predetermined height.Alternatively, for example, the image capturing devices 10A may bearranged at predetermined intervals hanging from the ceiling of theshowroom. The plurality of image capturing devices 10A are used tocapture images of the entire distribution site A. The distributionterminal 30A receives captured image data from the image capturingdevices 10A and distributes the received captured image data to thecommunication terminals 70 at viewing sites. The numbers of imagecapturing devices 10A and distribution terminals 30A arranged at thedistribution site A are not limited to those illustrated in FIG. 10.That is, as illustrated in FIG. 9, one image capturing device 10A andone distribution terminal 30A may be arranged at the distribution siteA. As illustrated in FIG. 10, many image capturing devices 10A and onedistribution terminal 30A may be arranged at the distribution site A.The same applies to the distribution site B.

In an existing system in which images captured by a plurality of imagecapturing devices arranged at a remote site are viewable at viewingsites, a user who desires to view the situation at a specific portion inthe distribution site does not know which of the images captured by theimage capturing devices to view. In addition, in a case in which usersat different viewing sites desire to view different portions in thedistribution site, operations on the image capturing devices (e.g.,pan-tilt-zoom (PTZ) operations) may conflict with each other. As aresult, it may be difficult to provide an exclusive viewing that allowsa plurality of users who view the same distribution site to viewdifferent portions. Further, even in a case in which spherical images,as described above, are captured by the image capturing devices, a userperforms individual operations on a plurality of spherical images toperform a process for displaying an intended portion, and it isdifficult for a user to perform an intuitive operation. To address thisinconvenience, the image communication system 1 allows a user to performan intuitive operation on a display screen such that a plurality ofusers are able to simultaneously view, with interest, different portionsin the distribution site using a captured image acquired from the sameimage capturing device.

Hardware Configuration of Image Communication System

Next, the hardware configuration of the devices or terminals of theimage communication system 1 according to an embodiment will bedescribed with reference to FIGS. 11 and 12. In the hardwareconfigurations illustrated in FIGS. 11 and 12, certain hardware elementsmay be added or deleted as appropriate.

Hardware Configuration of Image Capturing Device

First, the hardware configuration of the image capturing device 10 willbe described with reference to FIG. 11. FIG. 11 is a diagramillustrating an example hardware configuration of the image capturingdevice 10. In the following description, the image capturing device 10is a spherical (omnidirectional) image capturing device including twoimaging elements. However, the image capturing device 10 may include anysuitable number of imaging elements greater than or equal to two imagingelements. In addition, the image capturing device 10 is not necessarilya device dedicated to capturing of an omnidirectional image.Alternatively, a typical digital camera, smartphone, or the like may beequipped with an external omnidirectional image capturing unit toimplement an image capturing device having substantially the samefunctions as those of the image capturing device 10.

As illustrated in FIG. 11, the image capturing device 10 includes animaging device 101, an image processor 104, an imaging controller 105, amicrophone 108, an audio processor 109, a central processing unit (CPU)111, a read only memory (ROM) 112, a static random access memory (SRAM)113, a dynamic random access memory (DRAM) 114, the operation device115, an input/output interface (I/F) 116, a short-range communicationcircuit 117, an antenna 117 a of the short-range communication circuit117, an electronic compass 118, a gyro sensor 119, an accelerationsensor 120, and a network I/F 121.

The imaging device 101 includes wide-angle lenses (so-called fish-eyelenses) 102 a and 102 b, each having an angle of view greater than orequal to 180 degrees so as to form a hemispherical image. The imagingdevice 101 further includes the two imaging elements 103 a and 103 bcorresponding to the wide-angle lenses 102 a and 102 b, respectively.The imaging elements 103 a and 103 b each include an image sensor suchas a complementary metal oxide semiconductor (CMOS) sensor or acharge-coupled device (CCD) sensor, a timing generation circuit, and agroup of registers. Each of the image sensors converts an optical imageformed by a corresponding one of the wide-angle lenses 102 a and 102 binto an electric signal to output image data. Each of the timinggeneration circuits generates a horizontal or vertical synchronizationsignal, a pixel clock, and the like for a corresponding one of the imagesensors. Each of the groups of registers has set therein variouscommands, parameters, and the like to be used for operations of acorresponding one of the imaging elements 103 a and 103 b.

The imaging elements 103 a and 103 b of the imaging device 101 areconnected to the image processor 104 via respective parallel I/F buses.In addition, the imaging elements 103 a and 103 b of the imaging device101 are connected to the imaging controller 105 via respective serialI/F buses such as inter-integrated circuit (I2C) buses. The imageprocessor 104, the imaging controller 105, and the audio processor 109are connected to the CPU 111 via a bus 110. The ROM 112, the SRAM 113,the DRAM 114, the operation device 115, the input/output I/F 116, theshort-range communication circuit 117, the electronic compass 118, thegyro sensor 119, the acceleration sensor 120, and the network I/F 121are also connected to the bus 110.

The image processor 104 acquires respective pieces of image data outputfrom the imaging elements 103 a and 103 b via the parallel I/F buses andperforms predetermined processing on the pieces of image data.Thereafter, the image processor 104 combines the pieces of image data,which are subjected to the predetermined processing, to generate data ofan equirectangular projection image as illustrated in FIG. 3C.

The imaging controller 105 usually functions as a master device whilethe imaging elements 103 a and 103 b usually function as slave devices.The imaging controller 105 sets commands and the like in the groups ofregisters of the imaging elements 103 a and 103 b via the respective I2Cbuses. The imaging controller 105 receives various commands from the CPU111. Further, the imaging controller 105 acquires status data and thelike of the groups of registers of the imaging elements 103 a and 103 bvia the respective I2C buses. The imaging controller 105 sends theacquired status data and the like to the CPU 111.

The imaging controller 105 instructs the imaging elements 103 a and 103b to output image data at a time when a shutter button of the operationdevice 115 is pressed (or tapped). Such a pressing or tapping operationis hereinafter referred to simply as “operation”. In some cases, theimage capturing device 10 has a function of displaying a preview imageon a display (e.g., a display of an external terminal such as asmartphone that performs short-range communication with the imagecapturing device 10 through the short-range communication circuit 117)or displaying a moving image. In the case of displaying a moving image,the imaging elements 103 a and 103 b continuously output image data at apredetermined frame rate (frames per minute).

Further, as described below, the imaging controller 105 operates incooperation with the CPU 111 to also function as a synchronizationcontroller for synchronizing the time when the imaging element 103 aoutputs image data and the time when the imaging element 103 b outputsimage data. Although the image capturing device 10 does not include adisplay in this embodiment, the image capturing device 10 may include adisplay. The microphone 108 converts sounds into audio data (signal).The audio processor 109 acquires the audio data output from themicrophone 108 via an I/F bus and performs predetermined processing onthe audio data.

The CPU 111 controls the entire operation of the image capturing device10 and also performs certain processing. The ROM 112 stores variousprograms for the CPU 111. The SRAM 113 and the DRAM 114 each operate asa work memory to store programs to be executed by the CPU 111 or databeing processed. More specifically, the DRAM 114 stores image data beingprocessed by the image processor 104 or data of the equirectangularprojection image on which processing has been performed.

The operation device 115 generally refers to various operation keys, apower switch, a shutter button, a touch panel having both the displayand operation functions, and the like. The user operates the operationdevice 115 to input various image capturing modes, image capturingconditions, or the like.

The input/output I/F 116 generally refers to an interface circuit suchas a USB I/F that allows the image capturing device 10 to communicatewith an external medium such as a Secure Digital (SD) card or a personalcomputer. The input/output I/F 116 may be either wired or wireless. Thedata of the equirectangular projection image, which is stored in theDRAM 114, is stored in an external medium via the input/output I/F 116or transmitted to an external terminal (or apparatus) via theinput/output IF 116, as appropriate.

The short-range communication circuit 117 communicates with an externalterminal (or apparatus) via the antenna 117 a of the image capturingdevice 10 using short-range wireless communication technology such asnear-field communication (NFC), Bluetooth (registered trademark), orWi-Fi (registered trademark). The short-range communication circuit 117is capable of transmitting the data of the equirectangular projectionimage to an external terminal (or apparatus).

The electronic compass 118 calculates an orientation of the imagecapturing device from the Earth's magnetism and outputs orientationinformation. The orientation information is an example of relatedinformation (metadata) in compliance with Exchangeable Image File Format(EXIF) and is used for image processing such as image correction ofcaptured images. The related information also includes data such as thedate and time when the image is captured, and the data size of the imagedata. The gyro sensor 119 detects a change in angle of the imagecapturing device 10 (roll angle, pitch angle, and yaw angle) withmovement of the image capturing device 10. The change in angle is anexample of related information (metadata) in compliance with EXIF and isused for image processing such as image correction of captured images.The acceleration sensor 120 detects acceleration in three axialdirections. The image capturing device 10 calculates the position (anangle with respect to the direction of gravity) of the image capturingdevice 10, based on the acceleration detected by the acceleration sensor120. The image capturing device 10 provided with the acceleration sensor120 improves the accuracy of image correction. The network I/F 121 is aninterface for performing data communication using the communicationnetwork 100 such as the Internet.

Hardware Configuration of Distribution Terminal

FIG. 12 is a diagram illustrating an example hardware configuration ofthe distribution terminal 30. The hardware components of thedistribution terminal 30 are denoted by reference numerals in the 300 s.The distribution terminal 30 is constructed by a computer. Asillustrated in FIG. 12, the distribution terminal 30 includes a CPU 301,a ROM 302, a random access memory (RAM) 303, a hard disk (HD) 304, ahard disk drive (HDD) controller 305, a display 306, an external deviceconnection I/F 308, a network I/F 309, a bus line 310, a keyboard 311, apointing device 312, a digital versatile disk rewritable (DVD-RW) drive314, a media I/F 316, an audio input/output I/F 317, a microphone 318, aspeaker 319, and a short-range communication circuit 320.

The CPU 301 controls the entire operation of the distribution terminal30. The ROM 302 stores a program used for driving the CPU 301, such asan initial program loader (IPL). The RAM 303 is used as a work area forthe CPU 301. The HD 304 stores various data such as a program. The HDDcontroller 305 controls reading or writing of various data from or tothe HD 304 under the control of the CPU 301. The display 306 displaysvarious kinds of information such as a cursor, a menu, a window,characters, and an image. The display 306 is an example of a displaydevice. In one example, the display 306 is a touch panel displayprovided with an input device. The external device connection I/F 308 isan interface for connecting to various external devices. Examples of theexternal devices include, but are not limited to, a USB memory and aprinter. The network I/F 309 is an interface for performing datacommunication using the communication network 100. The bus line 310 isan address bus, a data bus, or the like for electrically connecting thehardware elements illustrated in FIG. 12, such as the CPU 301.

The keyboard 311 is a type of input device provided with a plurality ofkeys for inputting characters, numerals, various instructions, or thelike. The pointing device 312 is a type of input device for selecting orexecuting various instructions, selecting a processing target, or movinga cursor being displayed. The input device is not limited to thekeyboard 311 and the pointing device 312 and may be a touch panel, avoice input device, or the like. The DVD-RW drive 314 controls readingor writing of various data from or to a DVD-RW 313, which is an exampleof a removable recording medium. The removable recording medium is notlimited to the DVD-RW and may be a digital versatile disk recordable(DVD-R), a Blu-ray Disc (registered trademark), or the like. The mediaI/F 316 controls reading or writing (storing) of data from or to arecording medium 315 such as a flash memory. The microphone 318 is atype of built-in sound collector for receiving input sounds. The audioinput/output I/F 317 is a circuit that processes input and output of anaudio signal between the microphone 318 and the speaker 319 under thecontrol of the CPU 301. The short-range communication circuit 320 is acommunication circuit for performing communication with an externalterminal (or apparatus) using short-range wireless communicationtechnology such as NFC, Bluetooth, or Wi-Fi.

Hardware Configuration of Communication Management Apparatus

FIG. 12 is a diagram illustrating an example hardware configuration ofthe communication management apparatus 50. The hardware components ofthe communication management apparatus 50 are denoted by referencenumerals in the 500 s in parentheses. The communication managementapparatus 50 is constructed by a computer. As illustrated in FIG. 12,since the communication management apparatus 50 has a configurationsimilar to that of the distribution terminal 30, the description of thehardware components will be omitted.

Hardware Configuration of Communication Terminal

FIG. 12 is a diagram illustrating an example hardware configuration ofthe communication terminal 70. The hardware components of thecommunication terminal 70 are denoted by reference numerals in the 700 sin parentheses. The communication terminal 70 is constructed by acomputer. As illustrated in FIG. 12, since the communication terminal 70has a configuration similar to that of the distribution terminal 30, thedescription of the hardware components will be omitted.

Further, each of the programs described above may be recorded in a filein an installable or executable format on a computer-readable recordingmedium for distribution. Examples of the recording medium include acompact disc recordable (CD-R), a digital versatile disk (DVD), aBlu-ray Disc, an SD card, and a USB memory. In addition, the recordingmedium may be provided in the form of a program product to users withina certain country or outside that country. For example, in thecommunication management apparatus 50, a program according to anembodiment of the present disclosure is executed to implement acommunication management method according to an embodiment of thepresent disclosure.

Functional Configuration of Image Communication System

Next, the functional configuration of the image communication system 1according to an embodiment will be described with reference to FIGS. 13to 20B. FIGS. 13 and 14 are diagrams illustrating an example functionalconfiguration of the image communication system 1. FIGS. 13 and 14illustrate devices and terminals related to the processes or operationsdescribed below among the devices and terminals illustrated in FIG. 9.

Functional Configuration of Image Capturing Device

First, the functional configuration of the image capturing device 10will be described with reference to FIG. 13. The image capturing device10 includes a communication unit 11, an acceptance unit 12, an imagingunit 13, a sound collection unit 14, and a storing and reading unit 19.The communication unit 11, the acceptance unit 12, the imaging unit 13,the sound collection unit 14, and the storing and reading unit 19 arefunctions implemented by any one of the hardware elements illustrated inFIG. 11 operating in accordance with instructions from the CPU 11according to an image capturing device program loaded onto the DRAM 114from the SRAM 113. The image capturing device 10 further includes astorage unit 1000. The storage unit 1000 is constructed by the ROM 112,the SRAM 113, and the DRAM 114 illustrated in FIG. 11. The storage unit1000 stores the globally unique identifier (GUID) of the image capturingdevice 10.

The communication unit 11 is mainly implemented by processing performedby the CPU 111 and communicates various data or information to anotherapparatus or terminal. The communication unit 11 performs, for example,data communication with another apparatus or terminal through theshort-range communication circuit 117 using short-range wirelesscommunication technology. Further, the communication unit 11 performs,for example, data communication with another apparatus or terminalthrough the input/output I/F 116 via various cables or the like. Thecommunication unit 11 further performs data communication with anotherapparatus or terminal through the network I/F 121 via the communicationnetwork 100.

The acceptance unit 12 is mainly implemented by processing performed bythe CPU 111 on the operation device 115 and accepts various selectionsor inputs from a user. The imaging unit 13 is mainly implemented byprocessing performed by the CPU 111 on the imaging device 101, the imageprocessor 104, and the imaging controller 105 and captures an objectsuch as scenery to acquire captured image data. The sound collectionunit 14 is mainly implemented by processing performed by the CPU 111 onthe microphone 108 and the audio processor 109 and collects soundsaround the image capturing device 10.

The storing and reading unit 19 is mainly implemented by processingperformed by the CPU 111 and stores various data (or information) in thestorage unit 1000 or reads various data (or information) from thestorage unit 1000.

Functional Configuration of Distribution Terminal

Next, the functional configuration of the distribution terminal 30 willbe described with reference to FIG. 13. The distribution terminal 30includes a transmitting/receiving unit 31, an acceptance unit 32, animage and audio processing unit 33, a display control unit 34, adetermination unit 35, a creation unit 36, a communication unit 37, anda storing and reading unit 39. The transmitting/receiving unit 31, theacceptance unit 32, the image and audio processing unit 33, the displaycontrol unit 34, the determination unit 35, the creation unit 36, thecommunication unit 37, and the storing and reading unit 39 are functionsimplemented by any one of the hardware elements illustrated in FIG. 12operating in accordance with instructions from the CPU 301 according toa distribution terminal program loaded onto the RAM 303 from the HD 304.The distribution terminal 30 further includes a storage unit 3000. Thestorage unit 3000 is constructed by the ROM 302, the RAM 303, and the HD304 illustrated in FIG. 12.

The transmitting/receiving unit 31 is mainly implemented by processingperformed by the CPU 301 on the network I/F 309 and transmits andreceives various data or information to and from another apparatus orterminal via the communication network 100.

The acceptance unit 32 is mainly implemented by processing performed bythe CPU 301 on the keyboard 311 or the pointing device 312 and acceptsvarious selections or inputs from a user.

The image and audio processing unit 33 is mainly implemented byprocessing performed by the CPU 301 and performs image processing oncaptured image data acquired by the image capturing device 10 capturingan object. The image and audio processing unit 33 further performs audioprocessing on audio data of a voice signal, which is obtained byconverting the voice of the user using the microphone 318. For example,the image and audio processing unit 33 performs image processing oncaptured image data received from the image capturing device 10, basedon image type information such as the source name so that the displaycontrol unit 34 causes the display 306 to display an image.Specifically, when the image type information indicates the specialimage, the image and audio processing unit 33 converts the capturedimage data (e.g., data of hemispherical images as illustrated in FIGS.3A and 3B) into spherical image data as illustrated in FIG. 4B to createspherical image data. Further, the image and audio processing unit 33outputs a voice signal of audio data distributed from another terminalvia the communication management apparatus 50 to the speaker 319 andoutputs a voice from the speaker 319.

The display control unit 34 is mainly implemented by processingperformed by the CPU 301 and causes the display 306 to display variousimages, characters, or the like. The determination unit 35 isimplemented by processing performed by the CPU 301 and performs variousdeterminations. For example, the determination unit 35 determines theimage type of captured image data received from the image capturingdevice 10.

The creation unit 36 is mainly implemented by processing performed bythe CPU 301 and creates a source name, which is an example of the imagetype information, in accordance with a naming rule, based on the generalimage or special image (that is, the spherical image) determined by thedetermination unit 35. For example, if the determination unit 35determines that the image type is the general image, the creation unit36 creates the source name “Video” indicating the general image. Bycontrast, if the determination unit 35 determines that the image type isthe special image, the creation unit 36 creates the source name“Video_Omni” indicating the special image.

The communication unit 37 is mainly implemented by processing performedby the CPU 301 on the short-range communication circuit 320 andcommunicates with the communication unit 11 of the image capturingdevice 10 using short-range wireless communication technology such asNFC, Bluetooth, or WiFi. In the foregoing description, the communicationunit 37 and the transmitting/receiving unit 31 are configured asseparate communication units. In another example, the communication unit37 and the transmitting/receiving unit 31 may share a singlecommunication unit.

The storing and reading unit 39 is mainly implemented by processingperformed by the CPU 301 and stores various data (or information) in thestorage unit 3000 or reads various data (or information) from thestorage unit 3000.

Image Capturing Device Management Table

FIG. 15A is a conceptual diagram illustrating an example image capturingdevice management table. The storage unit 3000 includes an imagecapturing device management database (DB) 3001. The image capturingdevice management DB 3001 is made up of an image capturing devicemanagement table illustrated in FIG. 15A. The image capturing devicemanagement table stores and manages a vendor ID and a product ID in theGUID of an image capturing device capable of obtaining two hemisphericalimages from which a spherical image is generated. Examples of the GUIDinclude a vendor ID (VID) and a product ID (PID), which are used by aUSB device. The vendor ID and the product ID may be stored when thedistribution terminal 30 is shipped from the factory or may beadditionally stored after the distribution terminal 30 is shipped fromthe factory, for example.

Image Type Management Table

FIG. 15B is a conceptual diagram illustrating an example image typemanagement table. The storage unit 3000 includes an image typemanagement DB 3002. The image type management DB 3002 is made up of animage type management table illustrated in FIG. 15B. The image typemanagement table stores, for each image data ID, an Internet Protocol(IP) address of an image capturing device, which is an example of anaddress of an image capturing device, and a source name (image typeinformation) in association with each other. The image data ID is anexample of image data identification information for identifying imagedata of an image to be distributed. The IP address of the imagecapturing device indicates the IP address of the image capturing device10 that has captured the image data indicated by the associated imagedata ID. The source name is a name for specifying the image capturingdevice 10 that has captured the image data indicated by the associatedimage data ID, and is an example of image type information. The sourcename is a name created by the distribution terminal 30 in accordancewith a predetermined naming rule.

The illustrated example indicates that four image capturing deviceshaving the IP addresses “2.1.2.3”, “2.1.1.5”, “2.1.5.4”, and “2.1.5.6”have transmitted image data indicated by image data IDs “RS001”,“RS002”, “RS003”, and “RS004”, respectively. It is also indicated thatthe image types indicated by the source names of the four imagecapturing devices are “Video_Omni”, “Video_Omni”, “Video”, and “Video”,which indicate the image types “special image”, “special image”,“general image”, and “general image”, respectively. In this embodiment,the special image is the spherical image. The IP address is an exampleof address information, and the address information may be a MediaAccess Control (MAC) address, a terminal identification (ID), or thelike. While the IP address is a simplified representation of theInternet Protocol version 4 (IPv4) address, the IP address may be anInternet Protocol version 6 (IPv6) address. In addition, data other thanimage data may be managed in association with the image data ID.Examples of the data other than image data include audio data, anddocument data to be used to share the screen between the distributionsite and the viewing sites.

Functional Configuration of Communication Management Apparatus

Next, the functional configuration of the communication managementapparatus 50 will be described with reference to FIG. 14. Thecommunication management apparatus 50 includes a transmitting/receivingunit 51, a selection unit 53, a determination unit 55, a generation unit56, a distribution site management unit 57, and a storing and readingunit 59. The transmitting/receiving unit 51, the selection unit 53, thedetermination unit 55, the generation unit 56, the distribution sitemanagement unit 57, and the storing and reading unit 59 are functionsimplemented by any one of the hardware elements illustrated in FIG. 12operating in accordance with instructions from the CPU 501 according toa communication management apparatus program loaded onto the RAM 503from the HD 504. The communication management apparatus 50 furtherincludes a storage unit 5000. The storage unit 5000 is constructed bythe ROM 502, the RAM 503, and the HD 504 illustrated in FIG. 12.

The transmitting/receiving unit 51 is mainly implemented by processingperformed by the CPU 501 on the network I/F 509 and transmits andreceives various data or information to and from another apparatus viathe communication network 100.

The selection unit 53 is mainly implemented by processing performed bythe CPU 501 and selects a specific image capturing device 10 among aplurality of image capturing devices 10. The selection unit 53 selects aspecific image capturing device 10 among the plurality of imagecapturing devices 10, based on, for example, coordinate valuesindicating each of a plurality of points of interest Pn accepted by thecommunication terminal 70, where n is a natural number greater than orequal to 2, and arrangement position information (coordinate values)indicating arrangement positions of the plurality of image capturingdevices 10. The plurality of points of interest Pn are hereinaftersometimes referred to simply as “points of interest Pn” or “points ofinterest P”. In this embodiment, the points of interest Pn indicatedesignated positions in an area where a plurality of image capturingdevices are arranged. That is, each point of interest indicates adesignated position. In this embodiment, the selection unit 53 functionsas an example of selection means. In this embodiment, coordinate valuesare also referred to as “coordinate information”.

The determination unit 55 is mainly implemented by processing performedby the CPU 501 and performs various determinations. The selection unit53 is mainly implemented by processing performed by the CPU 501 andselects an image capturing device 10 in response to a request from thecommunication terminal 70. In this embodiment, the determination unit 55functions as an example of a determiner.

The generation unit 56 is mainly implemented by processing performed bythe CPU 501 and generates an image data ID and predetermined-areainformation. The generation unit 56 generates, for example,predetermined-area information. The predetermined-area informationindicates a predetermined area (e.g., the predetermined area Tillustrated in FIG. and the like) in an image captured by the imagecapturing device 10 selected by the selection unit 53. An image in whichthe entire captured image is displayed (e.g., the spherical image CEillustrated in FIG. 5 and the like) is also referred to as “entireimage”. In this embodiment, the generation unit 56 functions as anexample of generation means. Specifically, the generation unit 56generates identification information of the image capturing device 10closest to each of the plurality of points of interest Pn,designated-position coordinate information (coordinate values)indicating the coordinates of a corresponding one of the plurality ofpoints of interest Pn, and predetermined-area information indicating apredetermined area in which the designated position (i.e., thecorresponding one of the plurality of points of interest Pn) is viewedat the front. The generation unit 56 generates identificationinformation of the image capturing devices 10 closest to each of theplurality of points of interest Pn in a plurality of sub-areas obtainedby dividing an area where the plurality of image capturing devices 10are arranged into a predetermined number by one or more lines passingthrough a corresponding one of the plurality of points of interest Pn.Further, the generation unit 56 calculates predetermined-areainformation indicating a predetermined area in which a predeterminedpoint of interest among the plurality of points of interest Pndesignated by the user appears at the front (or center), in associationwith the coordinate values of the predetermined point of interest.

The distribution site management unit 57 is mainly implemented byprocessing performed by the CPU 501 and manages distribution siteinformation indicating the state of the distribution site.

The storing and reading unit 59 is mainly implemented by processingperformed by the CPU 501 and stores various data (or information) in thestorage unit 5000 or reads various data (or information) from thestorage unit 5000.

Session Management Table

FIG. 16A is a conceptual diagram illustrating an example sessionmanagement table. The storage unit 5000 includes a session management DB5001. The session management DB 5001 is made up of a session managementtable illustrated in FIG. 16A. The session management table stores, foreach session ID, a site ID and an IP address of a participantcommunication terminal in association with each other. The session ID isan example of session identification information for identifying acommunication session for implementing image communication. The sessionID is generated for each virtual floor. The session ID is also managedby the communication terminals 70 and is used when each of thecommunication terminals 70 selects a communication session. The site IDis an example of site identification information for identifying adistribution site. The IP address of the participant communicationterminal indicates the IP address of a communication terminal 70participating in a virtual floor indicated by the associated session ID.

Image Type Management Table

FIG. 16B is a conceptual diagram illustrating an example image typemanagement table. The storage unit 5000 includes an image typemanagement DB 5002. The image type management DB 5002 is made up of animage type management table illustrated in FIG. 16B. The image typemanagement table stores the information managed by the image typemanagement table illustrated in FIG. 15B and the same session ID as thesession ID managed in the session management table illustrated in FIG.16A in association with each other. The communication managementapparatus 50 stores an image data ID, an IP address of an imagecapturing device, and image type information, which are the same asthose stores in the distribution terminal 30 and the communicationterminal 70, because, for example, when a new communication terminal 70enters a virtual floor, the communication management apparatus 50transmits information including the image type information to acommunication terminal 70 that is already in video communication and thenew communication terminal 70, which has newly participated in the videocommunication. As a result, the communication terminal 70 that isalready in the video communication and the communication terminal 70that has newly participated in the video communication do not have totransmit and receive such information including the image typeinformation.

Predetermined-Area Management Table

FIG. 17 is a conceptual diagram illustrating an examplepredetermined-area management table. The storage unit 5000 includes apredetermined-area management DB 5003. The predetermined-area managementDB 5003 is made up of a predetermined-area management table illustratedin FIG. 17. The predetermined-area management table stores, for the IPaddress of each image capturing device (image source) from whichcaptured image data is transmitted, point-of-interest coordinate values(x, y, z) indicating the position of a point of interest, an IP addressof a communication terminal (image destination) to which the capturedimage data is transmitted, and predetermined-area information inassociation with each other. The predetermined-area informationindicates a parameter of a predetermined-area image currently displayedon the communication terminal to which the captured image data istransmitted. As illustrated in FIGS. 6A, 6B, and 7, thepredetermined-area information is a conversion parameter for convertingthe captured image into the image of the predetermined area T (thepredetermined-area image Q) in the captured image.

For example, the information managed in the first row of thepredetermined-area management table illustrated in FIG. 17 indicatesthat after the distribution terminal 30 receives captured image dataobtained by the image capturing device 10 having the IP address“2.1.2.3”, the captured image data is transmitted from the distributionterminal 30 to the communication terminal 70 having the IP address“1.2.1.3” via the communication management apparatus 50.

When predetermined-area information including IP addresses in the sameset as an already managed set of the IP address of an image capturingdevice from which captured image data is transmitted and the IP addressof a communication terminal to which the captured image data istransmitted is newly received by the transmitting/receiving unit 51, thestoring and reading unit 59 rewrites the already managedpredetermined-area information to the newly received predetermined-areainformation

Arrangement Information Management Table

FIG. 18 is a conceptual diagram illustrating an example arrangementinformation management table. The storage unit 5000 includes anarrangement information management DB 5004. The arrangement informationmanagement DB 5004 is made up of an arrangement information managementtable illustrated in FIG. 18. The arrangement information managementtable stores, for each site ID for identifying a distribution site, anIP address of an image capturing device 10 arranged in the distributionsite, coordinate values indicating the position at which the imagecapturing device 10 is arranged, and the arrangement direction of theimage capturing device 10 in association with each other. The coordinatevalues are coordinate values on the map of the distribution site wherethe image capturing device 10 is arranged. The arrangement direction ofthe image capturing device 10 indicates the direction (angle) on the mapin which the front surface of the image capturing device 10 faces. Thearrangement direction is set to a desired direction (angle) by theadministrator of the distribution terminal 30 or the distribution site.

Distribution-Site Management Table

FIG. 19 is a conceptual diagram illustrating an exampledistribution-site management table. The storage unit 5000 includes adistribution site management DB 5005. The distribution site managementDB 5005 is made up of a distribution-site management table illustratedin FIG. 19. The distribution-site management table stores, for each siteID for identifying a distribution site, a site name and distributionsite information in association with each other. The distribution siteinformation indicates the state of the distribution site. Thedistribution site information includes a uniform resource locator (URL)for accessing map image data indicating a map of the distribution siteand site coordinate information (coordinate values) indicating thecoordinate values of the distribution site. The communication terminal70 accesses the URL transmitted from the communication managementapparatus 50 to acquire the map image data and the site coordinateinformation of the distribution site. The URL is an example of storagedestination information. The storage destination information is notlimited to the URL and may be a uniform resource identifier (URI) or thelike.

Functional Configuration of Communication Terminal

Next, the functional configuration of the communication terminal 70 willbe described with reference to FIG. 14. The communication terminal 70includes a transmitting/receiving unit (acquisition unit) 71, anacceptance unit 72, an image and audio processing unit 73, a displaycontrol unit 74, a determination unit 75, a creation unit 76, apoint-of-interest specifying unit 77, a selection unit 81, a generationunit 82, and a storing and reading unit 79. The transmitting/receivingunit 71, the acceptance unit 72, the image and audio processing unit 73,the display control unit 74, the determination unit 75, the creationunit 76, the point-of-interest specifying unit 77, the selection unit81, the generation unit 82, and the storing and reading unit 79 arefunctions implemented by any one of the hardware elements illustrated inFIG. 12 operating in accordance with instructions from the CPU 701according to a communication terminal program loaded onto the RAM 703from the HD 704. The communication terminal 70 further includes astorage unit 7000. The storage unit 7000 is constructed by the ROM 702,the RAM 703, and the HD 704 illustrated in FIG. 12.

The transmitting/receiving unit (acquisition unit) 71 is mainlyimplemented by processing performed by the CPU 701 on the network I/F709 and transmits and receives various data or information to and fromanother apparatus or terminal via the communication network 100. Thetransmitting/receiving unit (acquisition unit) 71 receives, for example,captured image data distributed from the distribution terminal 30 viathe communication management apparatus 50. Further, thetransmitting/receiving unit (acquisition unit) 71 functions as, forexample, an acquisition unit and acquires identification information ofa shareable image capturing device capable of sharing images of any twopoints of interest among the plurality of points of interest Pn. Thetransmitting/receiving unit (acquisition unit) 71 further acquirespieces of predetermined-area information of the shareable imagecapturing device, each indicating a predetermined area in which acorresponding one of the two points of interest is viewed at the front.

The acceptance unit 72 is mainly implemented by processing performed bythe CPU 701 on the keyboard 711 or the pointing device 712 and acceptsvarious selections or inputs from a user. For example, the acceptanceunit 72 accepts input of a plurality of points of interest Pn, which aredesignated by the user, in a distribution site. In this embodiment, theacceptance unit 72 functions as an example of acceptance means.

The image and audio processing unit 73 is mainly implemented byprocessing performed by the CPU 701 and performs image processing oncaptured image data distributed from the distribution terminal 30. Theimage and audio processing unit 73 further performs audio processing onaudio data distributed from the distribution terminal 30. For example,to display on the display 706 an image of a predetermined areacorresponding to predetermined-area information received by thetransmitting/receiving unit (acquisition unit) 71, the image and audioprocessing unit 73 applies perspective projection conversion to thecaptured image (spherical image) using the predetermined-areainformation to generate a predetermined-area image corresponding to thepredetermined-area information. Further, the image and audio processingunit 73 outputs a voice signal of audio data distributed from thedistribution terminal 30 via the communication management apparatus 50to the speaker 719 and outputs a voice from the speaker 719.

The display control unit 74 is mainly implemented by processingperformed by the CPU 701 and causes the display 706 to display variousimages, characters, or the like. For example, the display control unit74 causes the display 706 to display the predetermined-area imagegenerated by the image and audio processing unit 73.

The determination unit 75 is implemented by processing performed by theCPU 701 and performs various determinations. In this embodiment, thedisplay control unit 74 functions as an example of display controlmeans.

The creation unit 76 is mainly implemented by processing performed bythe CPU 701 and implements functions similar to those of the creationunit 36. The point-of-interest specifying unit 77 is mainly implementedby processing performed by the CPU 701 and specifies a point of interestP in the distribution site where the image capturing devices 10 arearranged. For example, the point-of-interest specifying unit 77specifies the coordinate information of each of a plurality of points ofinterest Pn designated by the user on the map image of the distributionsite.

The selection unit 81 is mainly implemented by processing performed bythe CPU 701 and implements functions similar to those of the selectionunit 53. The generation unit 82 is mainly implemented by processingperformed by the CPU 701 and generates predetermined-area information.For example, the generation unit 82 generates predetermined-areainformation indicating a predetermined area in an image captured by animage capturing device 10 selected by the selection unit 81. Further,the generation unit 82 functions as, for example, an acquisition unitand acquires predetermined-area information. The predetermined-areainformation indicates a predetermined area including each of theplurality of points of interest Pn in an image captured by a specificimage capturing device 10 selected based on the coordinate informationof the plurality of points of interest Pn, which are accepted by theacceptance unit 72, and the arrangement positions of the image capturingdevices 10.

The storing and reading unit 79 is mainly implemented by processingperformed by the CPU 701 and stores various data (or information) in thestorage unit 7000 or reads various data (or information) from thestorage unit 7000.

Image Type Management Table

FIG. 20A is a conceptual diagram illustrating an example image typemanagement table. The storage unit 7000 includes an image typemanagement DB 7001. The image type management DB 7001 is made up of animage type management table illustrated in FIG. 20A. Since the imagetype management table has substantially the same data configuration asthat of the image type management table illustrated in FIG. 15B, thedescription thereof will be omitted.

Predetermined-Area Management Table

FIG. 20B is a conceptual diagram illustrating an examplepredetermined-area management table. The storage unit 7000 includes apredetermined-area management DB 7002. The predetermined-area managementDB 7002 is made up of a predetermined-area management table illustratedin FIG. 20B. The predetermined-area management table stores, for the IPaddress of each image capturing device that has obtained captured imagedata, point-of-interest coordinate values (x, y, z) indicating theposition of a point of interest, and predetermined-area information inassociation with each other. The predetermined-area informationindicates a currently displayed predetermined-area image. As illustratedin FIGS. 6A, 6B, and 7, the predetermined-area information is aconversion parameter for converting the captured image into the image ofthe predetermined area T (the predetermined-area image Q) in thecaptured image.

The relationship between a point of interest P and a plurality of imagecapturing devices 10 arranged at the distribution site A will now bedescribed with reference to FIGS. 21 to 27.

Case in which Two Points of Interest P are Set

FIG. 21 is a view schematically describing an example relationshipbetween two points of interest P and the image capturing devices 10. Forexample, at the distribution site A (area A), such as a showroom, in theimage communication system 1 illustrated in FIG. 10, the plurality ofimage capturing devices 10A are arranged regularly in a grid pattern, asillustrated in FIG. 21. Based on the arrangement of the plurality ofimage capturing devices 10A, the generation unit 56 generates atwo-dimensional arrangement image (map image) based on the coordinatevalues of each of the image capturing devices 10A, for example. The userwho uses the communication terminal 70 can operate a “point of interest”button 860 while referring to a tour image 830 corresponding to atwo-dimensional arrangement image displayed in a site display screen 820on the display 706 of the communication terminal 70, which will bedescribed below. In FIG. 21, points of interest P1 and P2 are set in thedistribution site A. The points of interest P1 and P2 are an example ofa plurality of designated positions.

Process for Selecting Image Capturing Devices near Point of Interest P1

FIG. 22 is a view schematically describing an example of installationdirections and azimuth angles of image capturing devices 10A near thepoint of interest P1. As in the example illustrated in FIG. 22, theselection unit 53 of the communication management apparatus 50 dividesthe area of the distribution site A into four areas, as indicated bybroken lines, from the point of interest P1 indicated by a black circlein a two-dimensional arrangement image (map image) generated in thecommunication management apparatus 50. The four areas are areas 1-1,1-2, 1-4, and 1-3, which are illustrated clockwise from top left in thetwo-dimensional arrangement image. The selection unit 53 selects imagecapturing devices 10A-1-1, 10A-1-2, 10A-1-3, and 10A-1-4, which arearranged at positions closest to the point of interest P1, from withinthe areas 1-1, 1-2, 1-3, and 1-4, respectively. The selection unit 53can select the image capturing devices 10A-1-1, 10A-1-2, 10A-1-3, and10A-1-4, which are arranged at positions closest to the point ofinterest P1, from within the respective areas by using a knowntechnique, for example, a formula for determining a distance between twopoints.

Process for Selecting Image Capturing Devices Near Point of Interest P2

FIG. 23 is a view schematically describing an example of installationdirections and azimuth angles of image capturing devices 10A near thepoint of interest P2. Also in FIG. 23, the selection unit 22 can use themethod illustrated in the example in FIG. 22 to select image capturingdevices 10A-2-1, 10A-2-2, 10A-2-3, and 10A-2-4, which are arranged atpositions closest to the point of interest P2, from within respectiveareas. As illustrated in the example in FIG. 23, the selection unit 53of the communication management apparatus 50 divides the area of thedistribution site A into four areas, as indicated by dot-and-dash lines,from the point of interest P2 indicated by another black circle in thetwo-dimensional arrangement image (map image) generated in thecommunication management apparatus 50. The four areas are areas 2-1,2-2, 2-4, and 2-3, which are illustrated clockwise from top left in thetwo-dimensional arrangement image. As in the case of the point ofinterest P1, the selection unit 53 selects the image capturing devices10A-2-1, 10A-2-2, 10A-2-3, and 10A-2-4, which are arranged at positionsclosest to the point of interest P2, from within the areas 2-1, 2-2,2-3, and 2-4, respectively.

Case in which Three Points of Interest P are Set

FIG. 24 is a view schematically describing an example relationshipbetween three points of interest P and the image capturing devices 10.Also in FIG. 24, the selection unit 23 can use the method illustrated inthe example in FIGS. 22 and 23 to select image capturing devices10A-3-1, 10A-3-2, 10A-3-3, and 10A-3-4, which are arranged at positionsclosest to a point of interest P3, from within respective areas. As inthe example illustrated in FIG. 24, the selection unit 53 of thecommunication management apparatus 50 divides the area of thedistribution site A into four areas, as indicated by finer dotted linesthan those for the point of interest P1, from the point of interest P3indicated by another black circle in the two-dimensional arrangementimage (map image) generated in the communication management apparatus50. The four areas are areas 3-1, 3-2, 3-4, and 3-3, which areillustrated clockwise from top left in the two-dimensional arrangementimage. As in the case of the points of interest P1 and P2, the selectionunit 53 selects the image capturing devices 10A-3-1, 10A-3-2, 10A-3-3,and 10A-3-4, which are arranged at positions closest to the point ofinterest P3, from within the areas 3-1, 3-2, 3-3, and 3-4, respectively.

By contrast, FIG. 25 is a view schematically describing an examplerelationship between the points of interest P1 and P2 and the imagecapturing devices 10 arranged in irregular positions. As illustrated inFIG. 25, the image capturing devices 10 may be arranged in irregularpositions at the distribution site A. The selection unit 53 can selectimage capturing devices 10A arranged at positions closest to each of thepoints of interest P1 and P2, from within respective areas in the waydescribed above with reference to FIGS. 21 to 24.

Process or Operation According to Embodiment Session ParticipationProcess

Next, a process or operation of the image communication system 1according to an embodiment will be described with reference to FIGS. 26to 42. In the following description, a captured image of thedistribution site A is distributed to the communication terminal 70, byway of example. However, similar processing is performed fordistribution of an image from any other distribution site such as thedistribution site B. First, a process for participating in a specificcommunication session will be described with reference to FIGS. 26 and27. FIG. 26 is a sequence diagram illustrating an example process forparticipating in a specific communication session in the imagecommunication system 1. FIG. 27 is a view illustrating an example screenfor selecting a communication session.

First, a user (e.g., the user C1) at the viewing site C performs anoperation of displaying a selection screen for selecting a communicationsession. In response to the acceptance unit 72 accepting the operationof displaying the selection screen, the display control unit 74 of thecommunication terminal 70C causes the display 706 to display a selectionscreen 800 illustrated in FIG. 27 (step S11). The selection screen 800illustrated in FIG. 27 displays selection buttons 810 a, 810 b, 810 c,etc. indicating floors A1, B1, B2, etc. to be selected, respectively.The selection button 810 a and the other selection buttons areassociated with respective session IDs.

When the user C1 selects a selection button for the desired virtualfloor that is a distribution site (here, the selection button 810 a),the acceptance unit 72 accepts selection of a communication session(step S12). Then, the transmitting/receiving unit (acquisition unit) 71transmits to the communication management apparatus 50 a participationrequest to participate in the communication session with thedistribution site (step S13). Thus, the transmitting/receiving unit 51of the communication management apparatus 50 receives the participationrequest. The participation request includes a session ID indicating thecommunication session for which selection is accepted in step S12, andthe IP address of the communication terminal 70C, which is the requestsender terminal.

Then, the storing and reading unit 59 of the communication managementapparatus 50 adds, in the session management table (the sessionmanagement DB 5001, see FIG. 16A), the IP address received in step S13to the “IP address of participant terminal” field for the record of thesame session ID as the session ID received in step S13 to perform aprocess for participating in the communication session (step S14).Further, the storing and reading unit 59 reads, from the sessionmanagement DB 5001, the site ID associated with the session ID of thecommunication session for which the participation process is performed.Then, the storing and reading unit 59 searches the distribution-sitemanagement table (the distribution site management DB 5005, see FIG. 19)using the site ID read in step S14 as a search key to read thedistribution site information associated with the same site ID as theread site ID (step S15). Then, the transmitting/receiving unit 51transmits a participation request response to the communication terminal70C (step S16). Thus, the transmitting/receiving unit (acquisition unit)71 of the communication terminal 70C receives the participation requestresponse. The participation request response includes the session IDreceived in step S13, the distribution site information read in stepS15, and a participation process result. The participation processresult includes the site ID read in step S14. In the following, adescription will be given of a case in which the participation processis successful. Through a process similar to the process illustrated inFIG. 26, the communication terminal 70D at the viewing site D performs aprocess for participating in a communication session.

Process for Managing Image Type Information

Next, a process for managing the image type information will bedescribed with reference to FIG. 28. FIG. 28 is a sequence diagramillustrating a process for managing image type information in the imagecommunication system 1.

First, when a user at the distribution site A connects the imagecapturing device 10A to the distribution terminal 30A, the storing andreading unit 19 of the image capturing device 10A reads the GUID of theimage capturing device 10A from the storage unit 1000. Then, thecommunication unit 11 of the image capturing device 10A transmits theGUID of the image capturing device 10A to the distribution terminal 30A(step S31). Thus, the communication unit 37 of the distribution terminal30A receives the GUID of the image capturing device 10A.

Then, the determination unit 35 of the distribution terminal 30Adetermines whether the same vendor ID and product ID as the vendor IDand product ID in the GUID received in step S31 are managed in the imagecapturing device management table (the image capturing device managementDB 3001, see FIG. 15A) to determine the image type (step S32).Specifically, if the same vendor ID and product ID are managed in theimage capturing device management DB 3001, the determination unit 35determines that the image capturing device 10A is configured to capturea special image (here, a spherical image). By contrast, if the samevendor ID and product ID are not managed in the image capturing devicemanagement table (the image capturing device management DB 3001, seeFIG. 15A), the determination unit 35 determines that the image capturingdevice 10A is configured to capture a general image.

Then, the storing and reading unit 39 stores, in the image typemanagement table (the image type management DB 3002, see FIG. 15B), theIP address of the image capturing device 10A and image type informationindicating the determination result obtained in step S32 in associationwith each other (step S33). In this state, no image data ID isassociated with the IP address of the image capturing device 10A and theimage type information. The image type information is, for example, asource name determined in accordance with a predetermined naming rule oran image type (general image or special image).

Then, the transmitting/receiving unit 31 transmits to the communicationmanagement apparatus 50 an additional request for the image typeinformation (step S34). The additional request for the image typeinformation includes the IP address of the image capturing device 10Aand the image type information, which are stored in step S33, and thesite ID of the distribution site A. Thus, the transmitting/receivingunit 51 of the communication management apparatus 50 receives theadditional request for the image type information.

Then, the storing and reading unit 59 of the communication managementapparatus 50 searches the session management table (the sessionmanagement DB 5001, see FIG. 16A) using the site ID received in step S34as a search key to read the corresponding session ID (step S35).

Then, the generation unit 56 generates a unique image data ID (stepS36). Then, the storing and reading unit 59 stores, in the image typemanagement table (the image type management DB 5002, see FIG. 16B), thesession ID read in step S35, the image data ID generated in step S36,and the IP address of the image capturing device 10A and the image typeinformation received in step S34 in association with each other as a newrecord (step S37). Then, the transmitting/receiving unit 51 transmitsthe image data ID generated in step S36 to the distribution terminal 30A(step S38). Thus, the transmitting/receiving unit 31 of the distributionterminal 30A receives the image data ID.

Then, the storing and reading unit 39 of the distribution terminal 30Astores, in the image type management table (the image type management DB3002, see FIG. 15B), the image data ID received in step S38 inassociation with the IP address of the image capturing device 10A andthe image type information stored in step S33 (step S39).

On the other hand, the transmitting/receiving unit 51 of thecommunication management apparatus 50 transmits an image typeinformation addition notification to the communication terminal 70C(step S40). Thus, the transmitting/receiving unit (acquisition unit) 71of the communication terminal 70C receives the image type informationaddition notification. The image type information addition notificationincludes the image data ID generated in step S36, and the IP address ofthe image capturing device 10A and the image type information stored instep S37.

Then, the storing and reading unit 79 of the communication terminal 70Cstores, in the image type management table (the image type management DB7001, see FIG. 20A), the image data ID, the IP address of the imagecapturing device 10A, and the image type information, which are receivedin step S40, in association with each other as a new record (step S41).Accordingly, the distribution terminal 30A and the communicationterminal 70C can share the same information in the image type managementDBs 3002 and 7001, respectively. The image type information additionnotification is also transmitted to another communication terminal,namely, the communication terminal 70D, and is stored in the image typemanagement DB 7001 of the communication terminal 70D.

Process for Communicating Captured Image Data

Next, a process for transmitting the captured image data and audio dataobtained at the distribution site A to the communication terminals 70(i.e., the communication terminals 70C and 70D) via the communicationmanagement apparatus 50 will be described with reference to FIG. 29.FIG. 29 is a sequence diagram illustrating an example process forcommunicating captured image data and audio data in the imagecommunication system 1. FIG. 29 illustrates an example in which capturedimage data acquired by one image capturing device 10 is distributed tothe communication terminals 70. However, similar processing is performedwhen a plurality of pieces of captured image data acquired by otherimage capturing devices 10 arranged in the distribution site aredistributed.

First, the communication unit 11 of the image capturing device 10Atransmits to the distribution terminal 30A captured image data acquiredby capturing an object or surroundings such as scenery and audio dataacquired by collecting sounds (step S51). Thus, the communication unit37 of the distribution terminal 30A receives the captured image data andthe audio data. In this case, since the image capturing device 10A iscapable of obtaining two hemispherical images from which a sphericalimage is generated, as illustrated in FIGS. 3A and 3B, the capturedimage data includes data of two hemispherical images.

Then, the transmitting/receiving unit 31 of the distribution terminal30A transmits to the communication management apparatus 50 the capturedimage data and the audio data sent from the image capturing device 10A(step S52). Thus, the transmitting/receiving unit 51 of thecommunication management apparatus 50 receives the captured image data,the audio data, and the image data ID. An image data ID for identifyingthe captured image data to be transmitted and received is alsotransmitted and received.

Then, the transmitting/receiving unit 51 of the communication managementapparatus 50 transmits the captured image data and the audio data to thecommunication terminals (the communication terminals 70C and 70D)participating in the same session as the session in which thedistribution terminal 30A is participating (steps S53 and S54). Thus,the transmitting/receiving unit (acquisition unit) 71 of each of thecommunication terminals 70C and 70D receives the captured image data andthe audio data. The image data ID for identifying the captured imagedata to be transmitted and received is also transmitted and received.

Process for Displaying Points of Interest P

Next, a process for displaying an image of a plurality of points ofinterest Pn in a distribution site, which are designated by a user at aviewing site, will be described with reference to FIGS. 30 to 42. FIG.30 is a sequence diagram illustrating an example process for displayingpoints of interest P in the image communication system 1. FIG. 30illustrates an example in which the communication terminal 70C at theviewing site C displays a captured image distributed from thedistribution terminal 30A. Similar processing is performed when thecaptured image is displayed on the communication terminal 70D at theviewing site D, which is another viewing site.

First, the display control unit 74 of the communication terminal 70Cuses the distribution site information received in step S16 to cause thedisplay 706 to display a site display screen 820 indicating thesituation of the distribution site A (step S71). FIG. 33 is a viewillustrating an example point-of-interest designation screen displayedon the communication terminal 70. The site display screen 820illustrated in FIG. 33 displays a tour image 830 indicating thesituation of a distribution site.

The site display screen 820 illustrated in FIG. 33 includes the tourimage 830 for viewing a spherical image of the distribution site toenable remote viewers to see the distribution site, a point-of-viewchange icon 850 for changing the point of view to a predetermined imagecapturing point in the tour image 830, a cross (or a star or any othersuitable mark) 855 indicating each point of interest P, a “point ofinterest” button 860, an “OK” button 870, and a “cancel” button 880. The“point of interest” button 860 is operated to display each point ofinterest P in the distribution site. The “OK” button 870 is operated toset selection of the points of interest P1, P2, etc. The “cancel” button880 is operated to terminate the viewing of the distribution site.

The tour image 830 is displayed using the map image data included in thedistribution site information received in step S16. The tour image 830is an image of the distribution site, which is captured by the imagecapturing device 10 in advance. The user C1 can operate thepoint-of-view change icon 850 to understand the general arrangement andthe like of the distribution site. Specifically, for example, the userC1 operates the point-of-view change icon 850 using a pointer p1 whileviewing the tour image 830. As a result, the user C1 is able to view adesired portion in the distribution site. The tour image 830 isdescribed herein as being displayed using the map image data included inthe distribution site information received in step S16. However, thetour image 830 may be configured to display captured image data of thedistribution site, which is received in real time.

Then, the user C1 operates the “point of interest” button 860 andoperates the point-of-view change icon 850 using the pointer p1 suchthat the display control unit 74 accepts the designation (input) of thepoint of interest P1. Then, the user C1 operates the point-of-viewchange icon 850 in a place different from the point of interest P1 usingthe pointer p1 such that the acceptance unit 72 accepts the designation(input) of the point of interest P2. As described above, in response tothe user C1 inputting the plurality of points of interest P1, P2, etc.and operating the “OK” button 870, the acceptance unit 72 accepts theselection of the plurality of points of interest Pn by the user C1 (stepS72). The acceptance unit 72 may accept the designation of the pluralityof points of interest Pn in response to a click or double-clickoperation of the user C1 using a mouse, which is an example of an inputdevice.

Then, the point-of-interest specifying unit 77 specifiespoint-of-interest coordinates, which are the coordinates of each of theplurality of points of interest Pn for which the designation is acceptedin step S72 (step S73). The point-of-interest coordinates specified bythe point-of-interest specifying unit 77 are the coordinates of thecentral point of the tour image 830 being displayed when the designationof the plurality of points of interest Pn is accepted in step S72. Thepoint-of-interest coordinates may be coordinates indicating the positionof the pointer p1 in the tour image 830.

The transmitting/receiving unit (acquisition unit) 71 transmits to thecommunication management apparatus 50 an image-capturing-deviceselection request indicating a request for selecting an image capturingdevice 10 (step S74). Thus, the transmitting/receiving unit 51 of thecommunication management apparatus 50 receives theimage-capturing-device selection request transmitted from thecommunication terminal 70C. The image-capturing-device selection requestincludes the point-of-interest coordinates specified in step S73 and thesite ID received in step S16.

In the image communication system 1 according to this embodiment, whenthe processing of step S74 described above is executed, anotherapparatus or the like may be present between the communication terminal70C (communication terminal) and the communication management apparatus50 (communication management apparatus). That is, information may betransmitted and received between the communication terminal 70C and thecommunication management apparatus 50 via another apparatus or the like.

Process for Selecting Image Capturing Device

Then, the communication management apparatus 50 executes a process forselecting an image capturing device 10 in response to theimage-capturing-device selection request received in step S74 (stepS75). The process in step S75 will be described in detail with referenceto FIGS. 31 and 21 to 24. FIG. 31 is a flowchart illustrating an exampleprocess for selecting an image capturing device.

First, the storing and reading unit 59 searches the distribution-sitemanagement table (the distribution site management DB 5005, see FIG. 19)using the site ID received in step S74 as a search key to read thedistribution site information associated with the same site ID as thereceived site ID (step S101).

Then, the selection unit 53 divides, based on the point-of-interestcoordinates indicating each of the plurality of points of interest Pnreceived in step S74, the area of the distribution site A into aplurality of areas by a line passing through the point of interest P,which indicates a predetermined position (step S102). In the exampleillustrated in FIG. 22, the point of interest P1 designated by the userC1 is indicated by a black circle. In this case, the selection unit 53can select the image capturing devices 10A-1-1, 10A-1-2, 10A-1-3, and10A-1-4, which are arranged at positions closest to the point ofinterest P1, from within the respective areas. The same applies to thepoint of interest P2.

Then, the storing and reading unit 59 searches the arrangementinformation management table (the arrangement information management DB5004, see FIG. 18) using the site ID received in step S74 as a searchkey to read the arrangement information associated with the same site IDas the received site ID (step S103).

Then, the selection unit 53 selects the image capturing device 10closest to the point-of-interest coordinates for each area obtained as aresult of the division in step S102, based on the coordinate valuesindicated by the arrangement information read in step S103 (step S104).Specifically, the selection unit 53 sequentially performs processing onthe image capturing devices 10 indicated in the arrangement informationone by one.

First, for example, the selection unit 53 refers to the coordinatevalues indicated in the read arrangement information and determines inwhich of the areas (e.g., areas 1 to 4) the image capturing device 10 islocated. Thereafter, the selection unit 53 calculates the distancebetween the coordinate values of the image capturing device 10 and thepoint-of-interest coordinates of each point of interest. If the imagecapturing device 10 is the first image capturing device 10 in thedetermined area or if the image capturing device 10 has a shorterdistance than the image capturing device 10, which has been determinedto have “the shortest distance at present” in the determined area, theselection unit 53 determines that the image capturing device 10 is theimage capturing device 10 having “the shortest distance at present” inthe determined area, and holds the identification information of theimage capturing device 10 (e.g., the IP address of the image capturingdevice 10) and the calculated distance.

The selection unit 53 performs the process described above on all of theimage capturing devices 10 indicated in the arrangement information toselect the image capturing device 10 having “the shortest distance” ineach of the areas obtained as a result of the division. The selectionunit 53 selects a number of image capturing devices 10 equal to thenumber of areas obtained as a result of the division. In the exampleillustrated in FIG. 22, the selection unit 53 selects the imagecapturing devices 10A-1-1, 10A-1-2, 10A-1-3, and 10A-1-4 as the imagecapturing devices 10 closest to the point-of-interest coordinates of thepoint of interest P1 in the areas 1-1, 1-2, 1-3, and 1-4, respectively.Likewise, in the example illustrated in FIG. 23, the selection unit 53selects the image capturing devices 10A-2-1, 10A-2-2, 10A-1-3, and10A-2-4 as the image capturing devices 10 closest to thepoint-of-interest coordinates of the point of interest P2 in the areas2-1, 2-2, 2-3, and to 2-4, respectively. The process described above isalso performed in the case illustrated in FIG. 24 in which three pointsof interest are set.

Then, the selection unit 53 selects a shareable image capturing device,based on the coordinate values indicated by the arrangement informationread in step S103 (step S105).

Process for Selecting Shareable Image Capturing Device

FIG. 32 is a flowchart illustrating an example process for selecting ashareable image capturing device. First, the selection unit 53 selectstwo points of interest among the plurality of points of interest Pndesignated by the user C1 (step S151). In FIGS. 22 and 23, two points ofinterest, namely, the points of interest P1 and P2, are illustrated.

Then, the determination unit 55 determines whether a shareable imagecapturing device is present between the selected two points of interestP1 and P2 (step S152). The determination of whether a shareable imagecapturing device is present in step S152 is to determine whether animage capturing device is available that is capable of providingpredetermined-area information indicating a predetermined area in whicheach of the two designated positions (the points of interest P1 and P2)is viewed at the front. Specifically, the selection unit 53 reads fromthe predetermined-area management table (the predetermined-areamanagement DB 5003, see FIG. 17) coordinate values indicating theposition of each of the points of interest P1 and P2 designated by theuser C1. Then, the selection unit 53 searches the respective coordinatevalues of the image capturing devices 10, which are managed in thearrangement information management table (the arrangement informationmanagement DB 5004, see FIG. 18), and selects an image 10 for which thedistances from the coordinate values of the two points of interest(designated positions) are shortest and which is capable of sharing theimages of the two points of interest (designated positions). That is, animage capturing device 10 for which the distance from the point ofinterest P1 and the distance from the point of interest P2 are shortestis a shareable image capturing device.

If a shareable image capturing device is present between the selectedtwo points of interest (P1 and P2) (YES in step S152), the selectionunit 53 selects one shareable image capturing device. In the exampleillustrated in FIGS. 22 and 23, the selection unit 53 selects the imagecapturing device 10A-1-4 (the image capturing device 10A-2-1) as ashareable image capturing device for the points of interest P1 and P2(step S153). Since the calculation formula for determining shortestdistances from the coordinate values of two points of interest(designated positions) described above can be obtained using knowncomputation, the description thereof will be omitted.

Then, the determination unit 55 determines whether the determination ofwhether a shareable image capturing device is present is completed forall of the points of interest (step S154). The processing of step S154is performed because three or more points of interest may be designatedby the user C1. FIG. 24 is a view schematically describing an examplerelationship between three points of interest P and the image capturingdevices 10. In the example illustrated in FIG. 24, three points ofinterest, further including the point of interest P3, are designated bythe user C1. In a case in which two points of interest are designated,the determination of whether a shareable image capturing device ispresent is performed once. In a case in which three points of interestare designated, however, three combinations of two points of interestare generated, and the determination is performed three timesaccordingly. For example, the determination unit 55 first determineswhether a shareable image capturing device is present between the pointsof interest P1 and P2. Also, the determination unit 55 determineswhether a shareable image capturing device is present between the pointsof interest P1 and P3, and further determines whether a shareable imagecapturing device is present between the points of interest P2 and P3.Also in a case in which the user C1 designates four or more points ofinterest, the determination unit 55 performs the determination aplurality of times.

As in FIG. 24, if three points of interest P are designated and thedetermination unit 55 determines that the determination of whether ashareable image capturing device is present is not performed on all ofthe points of interest (NO in step S154), the process returns to stepS151. On the other hand, if two points of interest are designated or ifthree points of interest are designated and the determination of whethera shareable image capturing device is present is completed for all ofthe combinations of two points of interest (YES in step S154), theprocess exits. That is, if the determination unit 55 determines thatthere is a plurality of combinations of point-of-interest coordinates oftwo points of interest among the point-of-interest coordinatesindicating the plurality of points of interest Pn received by thetransmitting/receiving unit 51, the generation unit 56 generatesidentification information of a shareable image capturing deviceobtained for each of the plurality of combinations, and pieces ofpredetermined-area information of the shareable image capturing device,each indicating a predetermined area in which a corresponding one of thetwo designated positions in each combination is viewed at the front.

If there is no shareable image capturing device between the selected twopoints of interest (P1 and P2) (NO in step S152), the selection unit 53does not select a shareable image capturing device, but selects theimage capturing devices 10 closest to the point-of-interest coordinatesfor the respective areas obtained as a result of the division in stepS102 (step S155). In the example illustrated in FIG. 24, the selectionunit 53 selects the image capturing devices 10A-2-1, 10A-2-2, 10A-2-3,and 10A-2-4 arranged at positions closest to the point of interest P2from within the respective areas, and selects the image capturingdevices 10A-3-1, 10A-3-2, 10A-3-3, and 10A-3-4 arranged at positionsclosest to the point of interest P3 from within the respective areas.Then, the process proceeds to step S154.

FIG. 25 is a view schematically describing an example relationshipbetween the points of interest P1 and P2 and the image capturing devices10 arranged in irregular positions. The process for selecting ashareable image capturing device is also applicable to a condition, asillustrated in FIG. 25, in which the image capturing devices 10 arearranged at irregular (or discrete) intervals in the distribution site.

Referring back to FIG. 31, the generation unit 56 generatespredetermined-area information of the selected shareable image capturingdevice (step S106). In the processing of step S106, the generation unit56 generates predetermined-area information, based on the relationshipbetween predetermined-area information and the image of thepredetermined area T illustrated in FIG. 7. As described above, thegeneration unit 56 calculates predetermined-area information indicatinga predetermined area in which each point of interest is displayed at thefront (or center), based on the following information: thepoint-of-interest coordinate values and the predetermined-areainformation (the radial distance (r), the polar angle (θ), and theazimuth angle (ϕ)) corresponding to the point-of-interest coordinatevalues, which are managed in the predetermined-area management table(the predetermined-area management DB 5003, see FIG. 17); and thecoordinate values (x, y, z) corresponding to the IP address of theshareable image capturing device and the arrangement direction (theangle relative to the x axis) of the shareable image capturing device,which are managed in the arrangement information management table (thearrangement information management DB 5004, see FIG. 18). Specifically,the generation unit 56 generates predetermined-area information fordisplaying a predetermined-area image (an image in perspectiveprojection) such that the point of interest indicated by thepoint-of-interest coordinates is located at the center of a sphericalimage captured by the selected shareable image capturing device 10. Itis assumed here that the radial distance (r) and the polar angle (θ)have predetermined values set in advance. The use of the predeterminedvalues indicates that the image in perspective projection is processedwith the angle of view and the elevation angle thereof being constant.As illustrated in FIG. 21, the azimuth angle (ϕ) can be calculated as arelative value between the installation direction of the shareable imagecapturing device 10 indicated in the installation information and thedirection from the shareable image capturing device 10 to the point ofinterest indicated by the point-of-interest coordinates. Then, thegeneration unit 56 generates, based on the calculated results,predetermined-area information for displaying a predetermined-area image(displaying an image in perspective projection) centered on thepoint-of-interest coordinates from a spherical image acquired by theshareable image capturing device 10.

Then, the generation unit 56 generates predetermined-area information ofthe other image capturing devices selected for each point of interest(step S107). That is, the generation unit 56 generates pieces ofpredetermined-area information, each indicating a predetermined area inwhich a corresponding one of the plurality of points of interest Pn isviewed at the front, based on identification information of imagecapturing devices 10 closest to the corresponding one of the pluralityof points of interest Pn in a plurality of sub-areas obtained bydividing the area where the plurality of image capturing devices 10 arearranged into a predetermined number by one or more lines passingthrough the corresponding one of the plurality of points of interest Pn,and point-of-interest coordinate information indicating the coordinatesof the corresponding one of the plurality of points of interest Pn.Specifically, the generation unit 56 generates predetermined-areainformation for displaying a predetermined-area image (an image inperspective projection) such that the point of interest indicated by thepoint-of-interest coordinates is located at the front of a sphericalimage captured by each of the selected image capturing devices 10A. Thepredetermined-area information of the other image capturing devices isgenerated using a method similar to that in step S106 described above.The generation unit 56 executes the process described above on each ofthe image capturing devices 10A selected in step S104. The arrangementpositions indicating the coordinate values of the image capturingdevices 10A and the arrangement directions of the image capturingdevices 10A indicating the directions in which the front surfaces of theimage capturing devices 10A face are set in advance in the arrangementinformation management DB 5004 by the administrator or the like. In theprocessing of step S106 described above, to obtain a combination of theIP address of an image capturing device and predetermined-areainformation, the storing and reading unit 59 and the selection unit 53use the predetermined-area management table (the predetermined-areamanagement DB 5003, see FIG. 17) and the arrangement informationmanagement table (the arrangement information management DB 5004, seeFIG. 18), which are managed by the communication management apparatus50, to generate corresponding predetermined-area information.

Then, the storing and reading unit 59 stores the predetermined-areainformation generated in steps S106 and S107 in the predetermined-areamanagement table (the predetermined-area management DB 5003, see FIG.17) in association with the IP addresses of the corresponding imagecapturing devices 10A (step S108).

As described above, the communication management apparatus 50 can selecta shareable image capturing device 10 capable of providingpredetermined-area information for the plurality of points of interestPn, based on the respective point-of-interest coordinate values of theplurality of points of interest Pn transmitted from the communicationterminal 70C and the arrangement information (coordinate values) of theplurality of image capturing devices 10.

In this embodiment, the generation unit 56 functions as an example ofgeneration means. Further, the generation unit 56 that generatespredetermined-area information may be included in, instead of thecommunication management apparatus 50, another apparatus disposedbetween the communication management apparatus 50 and the communicationterminal 70.

Referring back to FIG. 30, the transmitting/receiving unit 51 of thecommunication management apparatus 50 transmits to the communicationterminal 70C a selection result notification that is a result of theprocess in step S75 (step S76). Thus, the transmitting/receiving unit(acquisition unit) 71 of the communication terminal 70C receives theselection result notification transmitted from the communicationmanagement apparatus 50. The selection result notification includesidentification information of the image capturing devices closest toeach of the points of interest Pn, including the shareable imagecapturing device, which are selected in step S104, the pieces ofpredetermined-area information generated in steps S106 and S107 suchthat, for each of the image capturing devices closest to each of thepoints of interest Pn, the point of interest appears at the front, andrespective pieces of predetermined-area information for the imagecapturing devices closest to each of the points of interest Pn. That is,the transmitting/receiving unit 51 transmits to the communicationterminal 70C identification information (IP addresses) of imagecapturing devices 10 closest to each of the plurality of points ofinterest Pn, which is generated by the generation unit 56, coordinateinformation (coordinate values) indicating the coordinates of each ofthe plurality of points of interest Pn, and pieces of predetermined-areainformation, each indicating a predetermined area in which acorresponding one of the plurality of points of interest Pn is viewed atthe front. As described above, the transmitting/receiving unit(acquisition unit) 71 of the communication terminal 70C acquires piecesof predetermined-area information, each indicating a predetermined areaincluding a corresponding one of the plurality of points of interest Pnfor which the selection is accepted in step S72.

As described above, the transmitting/receiving unit 51 transmits to thecommunication terminal 70C identification information (IP address) of ashareable image capturing device (the image capturing device 10A-1-4(10A-2-1)), selected from among a plurality of image capturing devices,for which distances from any two designated positions (the points ofinterest P1 and P2) among the designated positions (coordinate values)indicating the plurality of pieces of designated-position information(points of interest Pn) received by the transmitting/receiving unit 51are shortest and which is capable of sharing images of the twodesignated positions, and pieces of predetermined-area information ofthe shareable image capturing device (the image capturing device 10A-1-4(10A-2-1)), each indicating a predetermined area in which acorresponding one of the two designated positions is viewed at thefront.

The transmitting/receiving unit 51 of the communication managementapparatus 50 can transmit the pieces of predetermined-area informationto the communication terminal 70 at the same time or different times.When the transmitting/receiving unit 51 transmits the pieces ofpredetermined-area information to the communication terminal 70 atdifferent times, the pieces of predetermined-area information may betransmitted in any order.

Then, the storing and reading unit 79 of the communication terminal 70Cstores the predetermined-area information received in step S76 in thepredetermined-area management table (the predetermined-area managementDB 7002, see FIG. 20B) in association with the IP addresses of the imagecapturing devices 10 (step S77).

Then, the transmitting/receiving unit (acquisition unit) 71 transmits acaptured-image request to the communication management apparatus 50 toacquire an image captured by the selected image capturing device 10(step S78). Thus, the transmitting/receiving unit 51 of thecommunication management apparatus 50 receives the captured-imagerequest transmitted from the communication terminal 70. Thecaptured-image request includes the image data ID acquired in step S40.

Upon receipt of the captured-image request, the storing and reading unit59 of the communication management apparatus 50 searches the image typemanagement table (the image type management DB 5002, see FIG. 16B) usingthe image data ID as a search key to read the corresponding source name(image type information) (step S79). As a result, image datarepresenting the captured image corresponding to the image data ID isspecified.

The transmitting/receiving unit 51 transmits to the communicationterminal 70C the image data representing the captured image that is read(step S80). Thus, the transmitting/receiving unit 71 of thecommunication terminal 70C receives the image data transmitted from thecommunication management apparatus 50. The image data includes the imagedata ID of the captured image and audio data.

Then, in response to a predetermined input operation of the user C1, theacceptance unit 72 accepts selection of a display image to be displayedon the communication terminal 70C within the captured image datareceived in step S53 (step S81). In a case in which the communicationterminal 70C is capable of simultaneously displaying a plurality ofcaptured images or in a case in which fewer captured images than thenumber of images simultaneously displayable on the communicationterminal 70C are received, the processing of step S78 may be omitted.

Then, to display an image of a predetermined area specified in thepredetermined-area information corresponding to the display image forwhich selection is accepted in step S81, the image and audio processingunit 73 applies perspective projection conversion using thepredetermined-area information received in step S76 to generate apredetermined-area image (step S82). As a result, the communicationterminal 70C can generate a predetermined-area image including the pointof interest P designated by the user in an image (spherical image)captured by the image capturing device 10 selected by the communicationmanagement apparatus 50.

Then, the display control unit 74 causes the display 706 to display thepredetermined-area image generated in step S82 (step S83).

Example Predetermined-Area Images Displayed on Screen

Example screens displayed on the communication terminal 70 according tothis embodiment will now be described. FIGS. 34 to 37 are viewsillustrating example display screens on which images captured by imagecapturing devices near the point of interest P1 are displayed. In thecaptured images, an object (vehicle) is inclined by an angle (azimuthangle) between lines connecting the respective image capturing devicesand the point of interest P1 in FIG. 34, which is, for example, about 45degrees relative to the x axis. In the example illustrated in FIG. 34, apredetermined-area image X11 is displayed on the communication terminal70. In FIG. 34, the predetermined-area image X11 in which one vehicleappears in the background, which is captured when the image capturingdevice 10A-1-1 sees the point of interest P1 at the center whileinclined about 45 degrees counterclockwise from the x axis, is displayedon the communication terminal 70. Also in FIGS. 35, 36, and 37,predetermined-area images X12, X13, and X14 are displayed on thecommunication terminal 70, respectively.

FIGS. 38 to 41 are views illustrating example display screens on whichimages captured by image capturing devices near the point of interest P2are displayed. In the captured images, an object (vehicle) is inclinedby an angle (azimuth angle) between lines connecting the respectiveimage capturing devices and the point of interest P2 in FIG. 38, whichis, for example, about 60 degrees relative to the x axis. In the exampleillustrated in FIG. 38, a predetermined-area image X21 is displayed onthe communication terminal 70. In FIG. 38, the predetermined-area imageX21 in which another vehicle appears in the background, which iscaptured when the image capturing device 10A-2-1 sees the point ofinterest P2 at the center while inclined about 60 degreescounterclockwise from the x axis, is displayed on the communicationterminal 70. Also in FIGS. 39, 40, and 41, predetermined-area imagesX22, X23, and X24 are displayed on the communication terminal 70,respectively.

Example Screen Displayed on Communication Terminal

FIG. 42 is a view illustrating an example display screen on thecommunication terminal 70 in response to selection of a plurality ofpoints of interest. First, the user of the communication terminal 70operates the point-of-view change icon 850, the pointer p1, the “pointof interest” button 860, and the like in the tour image 830 (see FIG.33) displayed on the display 706 to designate the points of interest P1and P2 at two different positions. In response to receipt of thedesignation of the points of interest P1 and P2 by the user, the displaycontrol unit 74 of the communication terminal 70 displays a displayscreen 950 illustrated in FIG. 42. For example, the display screen 950presents an area-1 screen 960 including images captured by imagecapturing devices in area 1 provided by the point of interest P1designated by the user. The area-1 screen 960 includes screens 961, 962,963, and 964, each displaying a predetermined-area image in which thepoint of interest P1 is viewed at the center. The screens 961, 962, 963,and 964 are displayed in the display screen 950 such that the screens961, 962, 963, and 964 display predetermined-area images in which thepoint of interest P1 is viewed at the center in spherical imagescaptured by the image capturing devices 10A-1-1, 10A-1-2, 10A-1-3, and10A-1-4 closest to the point of interest P1, respectively. As describedabove, the image capturing devices 10A-1-1, 10A-1-2, 10A-1-3, and10A-1-4 closest to the point of interest P1 are selected by theselection unit 53 from within the areas 1-1, 1-2, 1-3, and 1-4, whichare obtained by dividing the area into four from the point of interestP1, respectively.

The display screen 950 also presents an area-2 screen 970 includingimages captured by image capturing devices in area 2 provided by thepoint of interest P2 designated by the user. The area-2 screen 970includes screens 971, 972, 973, and 974, each displaying apredetermined-area image in which the point of interest P2 is viewed atthe center. The screens 971, 972, 973, and 974 are displayed in thedisplay screen 950 such that the screens 971, 972, 973, and 974 displaypredetermined-area images in which the point of interest P2 is viewed atthe center in spherical images captured by the image capturing devices10A-2-1, 10A-2-2, 10A-2-3, and 10A-2-4 closest to the point of interestP2, respectively. As described above, the image capturing devices10A-2-1, 10A-2-2, 10A-2-3, and 10A-2-4 closest to the point of interestP2 are selected by the selection unit 53 from within the areas 2-1, 2-2,2-3, and 2-4, which are obtained by dividing the area into four from thepoint of interest P2, respectively.

In this case, the predetermined-area image in which the point ofinterest P1 is viewed at the center in the area 1-4 in the area-1 screen960 and the predetermined-area image in which the point of interest P2is viewed at the center in the area 2-1 in the area-2 screen 970 are twopredetermined-area images in the spherical image captured by the sameimage capturing device 10A-1-4 (or 10A-2-1). That is, as illustrated inFIG. 42, the user is able to simultaneously view differentpredetermined-area images in the spherical image captured by the sameimage capturing device 10A-1-4 (or 10A-2-1) arranged at the distributionsite. At this time, the display control unit 74 of the communicationterminal 70 may change the content displayed in at least one of thesub-screen 1-4 in the area-1 screen 960 and the sub-screen 2-1 in thearea-2 screen 970 to distinguish them from the sub-screens displayingthe other predetermined-area images and indicate that the sub-screens1-4 and 2-1 display predetermined-area images captured by the same imagecapturing device. Specific examples of the method for changing thecontent displayed in at least one of the sub-screens 1-4 and 2-1 includeadding a predetermined character, alphanumeric character, sign, mark,background, or the like, changing the screen display size, and makingthe screen blink. In the example illustrated in FIG. 42, a sun mark isadded to both the sub-screens 1-4 and 2-1 by the display control unit74.

In the display screen 950, a map 980 is further displayed by displaycontrol unit 74. The map 980 indicates a bird's-eye view of thedistribution site A. The map 980 contains the points of interest P1 andP2 designated by the user, which are indicated by black circles, at thecoordinates corresponding to the respective coordinate values of thepoints of interest P1 and P2. This enables the user to simultaneouslycheck the predetermined-area images distributed from the distributionsite for each point of interest designated by the user while comparingthe area-1 screen 960, the area-2 screen 970, and the map 980.

As described above, in the image communication system 1, image capturingdevices suitable to simultaneously view the plurality of points ofinterest Pn designated by a user at a viewing site can be selected usingthe communication management apparatus 50. In the image communicationsystem 1, furthermore, predetermined-area information capable ofdisplaying images of the plurality of points of interest Pn, which arecaptured by selected image capturing devices 10, is transmitted from thecommunication management apparatus 50 to the communication terminal 70.As a result, the images indicating the plurality of points of interestPn can be displayed on the communication terminal 70.

In the image communication system 1 according to this embodiment, whenprocessing such as the processing of steps S74 to S76 or the processingof steps S78 to S80 described above is executed, another apparatus orthe like may be present between the communication management apparatus50 (communication management apparatus) and the communication terminal70 (communication terminal). That is, information may be transmitted andreceived between the communication management apparatus 50 and thecommunication terminal 70 via another apparatus.

In this embodiment, the selection unit 53 divides the distribution siteinto four areas, by way of example. However, the selection unit 53 maychange the number of areas in accordance with the number of imagecapturing devices 10 or distribution terminals 30 arranged in thedistribution site, the network bandwidth of the communication network100, the number of images simultaneously receivable and displayable atthe communication terminal 70, or the like.

Modification of Process for Displaying Points of Interest P

Next, a modification of the process for displaying the points ofinterest P in the image communication system 1 will be described withreference to FIG. 43. FIG. 43 is a sequence diagram illustrating amodification of the process for displaying points of interest P in theimage communication system 1. FIG. 43 illustrates an example in whichthe process for selecting an image capturing device 10 described aboveis executed by the communication terminal 70C at the viewing site C.Since the processing of steps S201 to S203 is similar to the processingof steps S71 to S73 in FIG. 30, the description thereof will be omitted.

In FIG. 43, the transmitting/receiving unit (acquisition unit) 71transmits to the communication management apparatus 50 anarrangement-information acquisition request, which is a request foracquiring arrangement information indicating the arrangement positionsof the image capturing devices 10 (step S204). Thus, thetransmitting/receiving unit 51 of the communication management apparatus50 receives the arrangement-information acquisition request transmittedfrom the communication terminal 70C. The arrangement-informationacquisition request includes the site ID received in steps S14 and S16.

Then, the communication management apparatus 50 searches the arrangementinformation management table (the arrangement information management DB5004, see FIG. 18) using the site ID received in step S204 as a searchkey to read the arrangement information associated with the same site IDas the received site ID (step S205). Then, the transmitting/receivingunit 51 transmits the arrangement information read in step S205 to thecommunication terminal 70C (step S206). Thus, the transmitting/receivingunit (acquisition unit) 71 of the communication terminal 70C receivesthe arrangement information transmitted from the communicationmanagement apparatus 50.

Then, the selection unit 81 executes a process for selecting an imagecapturing device 10, based on the coordinate values indicated by thearrangement information read in step S206 (step S207). Since the processfor selecting an image capturing device 10 executed in step S207 issimilar to the processing of step S75 illustrated in FIG. 30 (thedetailed process is illustrated in the flowchart in FIG. 31), thedescription thereof will be omitted. The processing of steps S101 andS103 may be omitted. Then, in step S107, the storing and reading unit 79stores the predetermined-area information in the predetermined-areamanagement table (the predetermined-area management DB 7002, see FIG.20B) instead of the predetermined-area management table (thepredetermined-area management DB 5003, see FIG. 17). Since theprocessing of steps S208 to S213 is similar to the processing of stepsS78 to S83 in FIG. 30, the description thereof will be omitted.

As described above, in the image communication system 1 according to amodification of the embodiment, the communication terminal 70 performs aprocess for selecting an image capturing device 10 that captures imagesin which points of interest P designated by a user are displayed. As aresult, images indicating the points of interest P can be displayed onthe communication terminal 70. Therefore, as in the embodiment describedabove, the image communication system 1 allows a user at a viewing siteto simultaneously view images of a plurality of points of interestdesignated by the user.

An image capturing device capable of PTZ operations has a delay in itsresponse to PTZ operations and a limited mechanical operation range. Inparticular, considering the mechanical operation range, even if awide-angle camera capable of covering about 180° is placed in front of adesignated point of interest, an image capturing device capable of PTZoperations is moved with a movement of an object near the point ofinterest. It is therefore desirable that a plurality of image capturingdevices be arranged at a distribution site.

As described above, the communication management apparatus 50 in theimage communication system 1 according to an embodiment of the presentdisclosure receives coordinate information of a plurality of points ofinterest Pn transmitted from the communication terminal 70 (step S74),and transmits to the communication terminal 70 the IP address of theshareable image capturing device 10A-1-4 (10A-2-1) for which distancesfrom the coordinate values of any two points of interest (the respectivecoordinate values of the points of interest P1 and P2) among therespective coordinate values of the plurality of points of interest Pnare shortest and which is capable of sharing images of the two points ofinterest P1 and P2, and pieces of predetermined-area information of theshareable image capturing device 10A-1-4 (10A-2-1), each indicating apredetermined area in which a corresponding one of the two points ofinterest P1 and P2 is viewed at the front (step S76). The communicationterminal 70 receives the pieces of predetermined-area information of theshareable image capturing device 10A-1-4 (10A-2-1), each indicating apredetermined area in which a corresponding one of the two points ofinterest P1 and P2 is viewed at the front (step S76), and displays, onthe display 706, predetermined-area images in which the two points ofinterest P1 and P2 are viewed at the front, based on the pieces ofpredetermined-area information of the shareable image capturing device10A-1-4 (10A-2-1) (step S83). As described above, the communicationmanagement apparatus 50 can cause the communication terminal 70 tosimultaneously display respective predetermined-area images of aplurality of points of interest Pn designated by a user, which areobtained by a shareable image capturing device, without performingspecial processing for the image capturing devices 10. This enables theuser who designates the plurality of points of interest Pn tosimultaneously check predetermined-area images of points of interest,which are captured by a single image capturing device, on the screen ofthe communication terminal 70.

In addition, since a shareable image capturing device is selected, theprocessing load on the communication management apparatus 50 in theimage communication system 1 may be reduced by an amount correspondingto one image capturing device.

In the embodiment described above, as an example of a spherical image(spherical panoramic image), the captured image (entire image) is athree-dimensional panoramic image. However, the captured image (entireimage) may be a two-dimensional panoramic image.

In the embodiment described above, furthermore, the communicationterminal 70 at a viewing site may not be dedicated to viewing. Thecommunication terminal 70 may be configured to distribute a capturedimage and simultaneously implement both the distribution function andthe viewing function. Likewise, the distribution terminal 30 at adistribution site may not be dedicated to distribution. The distributionterminal 30 may be configured to display a captured image distributedfrom any other site and simultaneously implement both the distributionfunction and the viewing function. As described above, the imagecommunication system 1 may be configured to perform two-waycommunication of a captured image between a plurality of sites.

Each of the functions in the embodiment described above may beimplemented by one or more processing circuits or circuitry. In theembodiment described above, the term “processing circuit or circuitry”includes a device programmed to implement each function using software,such as a processor implemented by an electronic circuit. Examples ofthe device include an application specific integrated circuit (ASIC), adigital signal processor (DSP), a field programmable gate array (FPGA),a system on a chip (SOC), a graphics processing unit (GPU), and aconventional circuit module.

Further, various tables in the embodiment described above may begenerated by machine learning. Further, data of associated items can beclassified, such that use of tables can be optional. As used herein,machine learning is a technique that enables a computer to acquirehuman-like learning ability. Machine learning refers to a technology inwhich a computer autonomously generates an algorithm to be used fordetermination such as data identification from learning data loaded inadvance and applies the generated algorithm to new data to make aprediction. Any suitable learning method is applied for machinelearning, for example, any one of supervised learning, unsupervisedlearning, semi-supervised learning, reinforcement learning, and deeplearning, or a combination of two or more of those learning methods.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention.

Any one of the above-described operations may be performed in variousother ways, for example, in an order different from the one describedabove.

1. A communication management apparatus for communicating with acommunication terminal that displays images captured by a plurality ofimage capturing devices, the communication management apparatuscomprising circuitry configured to: receive pieces ofdesignated-position information transmitted from the communicationterminal, each of the pieces of designated-position informationindicating a corresponding one of a plurality of designated positions inan area where the plurality of image capturing devices are provided; andtransmit, to the communication terminal, identification information of ashareable image capturing device selected from among the plurality ofimage capturing devices and pieces of predetermined-area information ofthe shareable image capturing device, the shareable image capturingdevice being an image capturing device for which distances from any twodesignated positions among the plurality of designated positionsindicated by the pieces of designated-position information that arereceived are shortest and which is capable of sharing images of the twodesignated positions, each of the pieces of predetermined-areainformation of the shareable image capturing device indicating apredetermined area in which a corresponding one of the two designatedpositions is viewed at the front.
 2. The communication managementapparatus according to claim 1, wherein a plurality of combinations ofany two designated positions among the plurality of designated positionsare present, and wherein the circuitry transmits, to the communicationterminal, identification information of a shareable image capturingdevice determined for each of the plurality of combinations, and piecesof predetermined-area information of the shareable image capturingdevice, each indicating a predetermined area in which a correspondingone of the two designated positions in the combination is viewed at thefront.
 3. The communication management apparatus according to claim 1,wherein the circuitry is further configured to select the shareableimage capturing device from among the plurality of image capturingdevices, based on coordinate information indicating each of theplurality of designated positions that is received, and arrangementposition information indicating arrangement positions of the pluralityof image capturing devices.
 4. The communication management apparatusaccording to claim 1, wherein the circuitry is further configured togenerate pieces of predetermined-area information, each indicating apredetermined area in which a corresponding one of the plurality ofdesignated positions is viewed at the front, the generation of each ofthe pieces of predetermined-area information being based onidentification information of image capturing devices closest to acorresponding one of the plurality of designated positions in aplurality of sub-areas obtained by dividing the area where the pluralityof image capturing devices are provided into a predetermined number byone or more lines passing through the corresponding one of the pluralityof designated positions, and designated-position coordinate informationindicating coordinates of the corresponding one of the plurality ofdesignated positions.
 5. The communication management apparatusaccording to claim 4, wherein the circuitry generates each of the piecesof predetermined-area information, based on arrangement directions andazimuth angles of image capturing devices closest to a corresponding oneof the plurality of designated positions in the plurality of sub-areas,the azimuth angles being angles relative to the corresponding one of theplurality of designated positions.
 6. The communication managementapparatus according to claim 4, wherein the circuitry transmits, to thecommunication terminal, identification information of an image capturingdevice closest to each of the plurality of designated positions,designated-position coordinate information indicating coordinates of acorresponding one of the plurality of designated positions, andpredetermined-area information indicating a predetermined area in whichthe corresponding one of the plurality of designated positions is viewedat the front.
 7. An image communication system comprising: thecommunication management apparatus according to claim 1; and acommunication terminal configured to display the images captured by theplurality of image capturing devices, the communication terminalincluding a terminal circuitry configured to: receive pieces ofpredetermined-area information, each indicating a predetermined area inwhich a corresponding one of the plurality of designated positions isviewed at the front; and control a display to display predetermined-areaimages based on the pieces of predetermined-area information, each ofthe predetermined-area images being an image in which a correspondingone of the plurality of designated positions is viewed at the front. 8.The image communication system according to claim 7, wherein theterminal circuitry of the communication terminal is further configuredto receive input of the plurality of designated positions in the areawhere the plurality of image capturing devices are provided, theplurality of designated positions being displayed on the display, andtransmit the pieces of designated-position information to thecommunication management apparatus, each of the pieces ofdesignated-position information indicating a corresponding one of theplurality of designated positions for which the input is received.
 9. Acommunication management method, performed by a communication managementapparatus configured to communicate with a communication terminal thatdisplays images captured by a plurality of image capturing devices, thecommunication management method comprising: receiving pieces ofdesignated-position information transmitted from the communicationterminal, each of the pieces of designated-position informationindicating a corresponding one of a plurality of designated positions inan area where the plurality of image capturing devices are provided; andtransmitting to the communication terminal identification information ofa shareable image capturing device selected from among the plurality ofimage capturing devices and pieces of predetermined-area information ofthe shareable image capturing device, the shareable image capturingdevice being an image capturing device for which distances from any twodesignated positions among the plurality of designated positionsindicated by the received pieces of designated-position information areshortest and which is capable of sharing images of the two designatedpositions, each of the pieces of predetermined-area information of theshareable image capturing device indicating a predetermined area inwhich a corresponding one of the two designated positions is viewed atthe front.
 10. Anon-transitory recording medium which, when executed byone or more processors in a communication management apparatus, causesthe processors to perform a communication management method, thecommunication management apparatus being configured to communicate witha communication terminal that displays images captured by a plurality ofimage capturing devices, the communication management method comprising:receiving pieces of designated-position information transmitted from thecommunication terminal, each of the pieces of designated-positioninformation indicating a corresponding one of a plurality of designatedpositions in an area where the plurality of image capturing devices areprovided; and transmitting to the communication terminal identificationinformation of a shareable image capturing device selected from amongthe plurality of image capturing devices and pieces ofpredetermined-area information of the shareable image capturing device,the shareable image capturing device being an image capturing device forwhich distances from any two designated positions among the plurality ofdesignated positions indicated by the received pieces ofdesignated-position information are shortest and which is capable ofsharing images of the two designated positions, each of the pieces ofpredetermined-area information of the shareable image capturing deviceindicating a predetermined area in which a corresponding one of the twodesignated positions is viewed at the front.